Introduction
Kaiba uses a configuration file to govern output structure and contents. This section is the introductionary course to Kaiba.
The setup¶
For this introduction course we will use kaiba-cli since it provides you with a simple command line tool to run kaiba. And no need to create any python files.
Install with pip:
pip install kaiba-cli
All examples will have a config, input and output json tab like this:
{}
{}
{}
Copy the contents of config.json and input.json down to your working dir.
Run all examples with the following unless otherwise stated.
kaiba config.json input.json
About JSON¶
Json is a human readable data format that stores data in objects consisting of attribute-value pairs and arrays. We will use the terms object and attribute quite often in this guide. To put it simply an object contains attributes that hold values. These values can sometimes be another object or even an array of objects.
{
"person": {
"name": "Bob",
"height": 180.5,
"friends": [
{
"name": "John",
"height": 170.5
}
]
}
}
person is an object, name and height are attribtes, "Bob" and 180.5 are values to those attributes. friends is a list(array) of objects.
The Root¶
The root of all ev... kaiba configs looks like this
{
"name": "root",
"attributes": [],
"objects": []
}
So this will fail since we consider empty result a failure, but this config generates the enclosing {} brackets you can see in the example in the About JSON section.
{}
Adding Attributes to root¶
To actually map some data we can add attributes.
{
"name": "root",
"attributes": [
{
"name": "firstname",
"default": "Thomas"
}
]
}
{}
{
"firstname": "Thomas"
}
Congratulations, you've just mapped a default value to an attribute! - Click output.json tab to see the output.
Structuring with objects¶
{
"name": "root",
"objects": [
{
"name": "person",
"attributes": [
{
"name": "firstname",
"default": "Thomas"
}
]
}
]
}
{}
{
"person": {
"firstname": "Thomas"
}
}
What we just did is the core principle of creating the output structure. We added an object with the name person, then we moved our firstname attribute to the person object.
Time to Fetch some values!¶
We will now introduce the data_fetchers key, it's and array of DataFetcher objects.
The DataFetcher object is the only place where you actually fetch data from the input. And you do that by specifying a path. The path describes the steps to take to get to the value we are interested in.
DataFetcher.path with flat structure¶
{
"name": "root",
"objects": [
{
"name": "person",
"attributes": [
{
"name": "firstname",
"data_fetchers": [
{
"path": ["name"]
}
]
}
]
}
]
}
{
"name": "Neo"
}
{
"person": {
"firstname": "Neo"
}
}
DataFetcher.path with nested structure¶
{
"name": "root",
"objects": [
{
"name": "actor",
"attributes": [
{
"name": "name",
"data_fetchers": [
{
"path": ["the_matrix", "neo", "actor", "name"]
}
]
}
]
}
]
}
{
"the_matrix": {
"neo": {
"actor": {
"name": "Keanu Reeves"
}
}
}
}
{
"actor": {
"name": "Keanu Reeves"
}
}
DataFetcher.path with data in lists¶
Consider the following json:
{
"data": ["Keanu", "Reeves", "The Matrix"]
}
In our DataFetcher object we supply path which is a list of how we get to our data. So how do we get the lastname in that data?
Easy, we reference the index of the list. The first data in the list starts at 0, second element 1, third 2 and so on. This number is the index and to get the last name we must use the index: 1
{
"name": "root",
"attributes": [
{
"name": "firstname",
"data_fetchers": [
{
"path": ["data", 0]
}
]
},
{
"name": "lastname",
"data_fetchers": [
{
"path": ["data", 1]
}
]
}
]
}
{
"data": ["Keanu", "Reeves", "The Matrix"]
}
{
"firstname": "Keanu",
"lastname": "Reeves"
}
Note
We still have to reference the "data" key first, so our path goes first to data then it finds the value at index 1
DataFetcher.path to list values and objects¶
Consider the following json:
{
"name": "neo",
"in_movies": [1, 2, 3, 4]
}
When we are actually interested in keeping the array in the output then we can simply supply the path to the list and it'll be outputted as is. This works for objects aswell.
{
"name": "root",
"attributes": [
{
"name": "character",
"data_fetchers": [
{
"path": ["name"]
}
]
},
{
"name": "plays_in_movies",
"data_fetchers": [
{
"path": ["in_movies"]
}
]
}
]
}
{
"name": "neo",
"in_movies": [1, 2, 3, 4]
}
{
"character": "neo",
"plays_in_movies": [
1,
2,
3,
4
]
}
Note
When fetching objects and arrays values some other functionality like casting obviously won't work.
Combining values¶
Now lets learn how to combine values from multiple places in the input.
It's fairly normal to only need name but getting firstname and lastname in input data. Lets combine them!
{
"name": "root",
"objects": [
{
"name": "actor",
"attributes": [
{
"name": "name",
"data_fetchers": [
{
"path": ["the_matrix", "neo", "actor", "firstname"]
},
{
"path": ["the_matrix", "neo", "actor", "lastname"]
}
],
"separator": " ",
}
]
}
]
}
{
"the_matrix": {
"neo": {
"actor": {
"firstname": "Keanu",
"lastname": "Reeves"
}
}
}
}
{
"actor": {
"name": "Keanu Reeves"
}
}
To find more values and combine them, simply add another DataFetcher object to data_fetchers array.
Use separator to control with what char values should be separated.
Regular Expressions¶
You can use Regex to find patterns in a given string and retrieve them as a string or as an array of strings.
Regex Example¶
Let's say you want to analyze your chess games, you get JSON with data BUT the fun part is inside pgn field, so let's retrieve Event, Site, Result, ECO and the game moves from pgn using Regex.
{
"name": "root",
"objects": [
{
"name": "game",
"attributes": [
{
"name": "event",
"data_fetchers": [
{
"path": ["pgn"],
"regex": {
"expression": "Event \\\"[\\w\\d ]+\\\""
},
"slicing": {
"from": 7,
"to": -1
}
}
]
},
{
"name": "site",
"data_fetchers": [
{
"path": ["pgn"],
"regex": {
"expression": "Site \\\"[\\w\\d. ]+\\\""
},
"slicing": {
"from": 6,
"to": -1
}
}
]
},
{
"name": "result",
"data_fetchers": [
{
"path": ["pgn"],
"regex": {
"expression": "Result \\\"[\\w\\d\/ -]+\\\""
},
"slicing": {
"from": 8,
"to": -1
}
}
]
},
{
"name": "eco",
"data_fetchers": [
{
"path": ["pgn"],
"regex": {
"expression": "ECO \\\"[\\w\\d ]+\\\""
},
"slicing": {
"from": 5,
"to": -1
}
}
]
},
{
"name": "moves",
"data_fetchers": [
{
"path": ["pgn"],
"regex": {
"expression": "\\s1\\..*"
},
"slicing": {
"from": 1
}
}
]
}
]
}
]
}
{
"black": {
"@id": "https://api.chess.com/pub/player/chameleoniasa",
"rating": 1273,
"result": "insufficient",
"username": "ChameleonIASA"
},
"end_time": 1347534562,
"fen": "8/8/8/8/7B/5k2/7K/8 b - -",
"pgn": "[Event \"Live Chess\"]\n[Site \"Chess.com\"]\n[Date \"2012.09.13\"]\n[Round \"-\"]\n[White \"GAURAV480\"]\n[Black \"ChameleonIASA\"]\n[Result \"1/2-1/2\"]\n[ECO \"A00\"]\n[ECOUrl \"https://www.chess.com/openings/Saragossa-Opening\"]\n[CurrentPosition \"8/8/8/8/7B/5k2/7K/8 b - -\"]\n[Timezone \"UTC\"]\n[UTCDate \"2012.09.13\"]\n[UTCTime \"11:03:43\"]\n[WhiteElo \"1283\"]\n[BlackElo \"1273\"]\n[TimeControl \"180\"]\n[Termination \"Game drawn by insufficient material\"]\n[StartTime \"11:03:43\"]\n[EndDate \"2012.09.13\"]\n[EndTime \"11:09:22\"]\n[Link \"https://www.chess.com/live/game/361066365\"]\n\n1. c3 {[%clk 0:03:00]} 1... e6 {[%clk 0:03:00]} 2. Qb3 {[%clk 0:02:57.1]} 2... Ne7 {[%clk 0:02:58.4]} 3. Nf3 {[%clk 0:02:47.7]} 3... d5 {[%clk 0:02:57.9]} 4. e3 {[%clk 0:02:45.6]} 4... Nd7 {[%clk 0:02:57.3]} 5. Be2 {[%clk 0:02:42.3]} 5... a6 {[%clk 0:02:56.9]} 6. O-O {[%clk 0:02:36.6]} 6... b5 {[%clk 0:02:56.3]} 7. d4 {[%clk 0:02:35.1]} 7... Bb7 {[%clk 0:02:55.7]} 8. Qc2 {[%clk 0:02:28.4]} 8... g6 {[%clk 0:02:55.3]} 9. Nbd2 {[%clk 0:02:25.4]} 9... Bg7 {[%clk 0:02:54.6]} 10. Nb3 {[%clk 0:02:23.6]} 10... c6 {[%clk 0:02:54.2]} 11. a3 {[%clk 0:02:21.3]} 11... a5 {[%clk 0:02:52]} 12. a4 {[%clk 0:02:17.9]} 12... bxa4 {[%clk 0:02:49.7]} 13. Rxa4 {[%clk 0:02:16.2]} 13... Nb6 {[%clk 0:02:45.5]} 14. Rxa5 {[%clk 0:02:12.4]} 14... Rxa5 {[%clk 0:02:43.9]} 15. Nxa5 {[%clk 0:02:11.4]} 15... Nc4 {[%clk 0:02:40.4]} 16. Nxb7 {[%clk 0:02:07.8]} 16... Qb6 {[%clk 0:02:36.9]} 17. Bxc4 {[%clk 0:01:59.4]} 17... dxc4 {[%clk 0:02:34.1]} 18. Nd6+ {[%clk 0:01:58.1]} 18... Kd7 {[%clk 0:02:30.1]} 19. Nxc4 {[%clk 0:01:55.2]} 19... Qa6 {[%clk 0:02:28.6]} 20. b4 {[%clk 0:01:49.9]} 20... Ra8 {[%clk 0:02:24.4]} 21. Qd3 {[%clk 0:01:45]} 21... Nd5 {[%clk 0:02:14.7]} 22. Nce5+ {[%clk 0:01:42.2]} 22... Bxe5 {[%clk 0:02:12.2]} 23. Nxe5+ {[%clk 0:01:37.6]} 23... Ke7 {[%clk 0:02:11.1]} 24. Qxa6 {[%clk 0:01:34.8]} 24... Rxa6 {[%clk 0:02:08.5]} 25. c4 {[%clk 0:01:32.9]} 25... Nxb4 {[%clk 0:02:05]} 26. Bd2 {[%clk 0:01:32]} 26... Nc2 {[%clk 0:01:59.5]} 27. h3 {[%clk 0:01:30.3]} 27... Kf6 {[%clk 0:01:55.7]} 28. Rb1 {[%clk 0:01:28.2]} 28... c5 {[%clk 0:01:43.9]} 29. dxc5 {[%clk 0:01:26.3]} 29... Kxe5 {[%clk 0:01:41.1]} 30. Bc3+ {[%clk 0:01:22.3]} 30... Ke4 {[%clk 0:01:39.6]} 31. Rb7 {[%clk 0:01:16.4]} 31... Kd3 {[%clk 0:01:37.9]} 32. Be5 {[%clk 0:01:11.9]} 32... Kxc4 {[%clk 0:01:35.7]} 33. Rxf7 {[%clk 0:01:07.8]} 33... Kxc5 {[%clk 0:01:34.2]} 34. Rxh7 {[%clk 0:01:06.7]} 34... Kd5 {[%clk 0:01:32.4]} 35. Bg3 {[%clk 0:01:03.4]} 35... Ra1+ {[%clk 0:01:28.5]} 36. Kh2 {[%clk 0:00:59.9]} 36... e5 {[%clk 0:01:24.4]} 37. Rh6 {[%clk 0:00:55.1]} 37... g5 {[%clk 0:01:21.8]} 38. Rh5 {[%clk 0:00:52.4]} 38... Ke4 {[%clk 0:01:17.7]} 39. Rxg5 {[%clk 0:00:51.4]} 39... Nb4 {[%clk 0:01:11.7]} 40. Rxe5+ {[%clk 0:00:48.3]} 40... Kd3 {[%clk 0:01:09.9]} 41. Rb5 {[%clk 0:00:44.8]} 41... Kc4 {[%clk 0:01:06.8]} 42. Rb8 {[%clk 0:00:39.9]} 42... Nd3 {[%clk 0:01:02.3]} 43. e4 {[%clk 0:00:35.4]} 43... Kd4 {[%clk 0:00:58.8]} 44. e5 {[%clk 0:00:33.6]} 44... Nxf2 {[%clk 0:00:53]} 45. Bxf2+ {[%clk 0:00:31]} 45... Kxe5 {[%clk 0:00:52.1]} 46. Rd8 {[%clk 0:00:23.4]} 46... Kf5 {[%clk 0:00:51.2]} 47. g4+ {[%clk 0:00:22.2]} 47... Kf4 {[%clk 0:00:49.1]} 48. h4 {[%clk 0:00:20.5]} 48... Kxg4 {[%clk 0:00:47.5]} 49. Rd4+ {[%clk 0:00:19.4]} 49... Kf3 {[%clk 0:00:44.7]} 50. h5 {[%clk 0:00:17.5]} 50... Ra6 {[%clk 0:00:41.5]} 51. h6 {[%clk 0:00:15.4]} 51... Rxh6+ {[%clk 0:00:37.6]} 52. Rh4 {[%clk 0:00:13.9]} 52... Rxh4+ {[%clk 0:00:34.9]} 53. Bxh4 {[%clk 0:00:12.8]} 1/2-1/2",
"rated": true,
"rules": "chess",
"time_class": "blitz",
"time_control": "180",
"url": "https://www.chess.com/live/game/361066365",
"white": {
"@id": "https://api.chess.com/pub/player/gaurav480",
"rating": 1283,
"result": "insufficient",
"username": "GAURAV480"
}
}
{
"game": {
"event": "Live Chess",
"site": "Chess.com",
"result": "1/2-1/2",
"eco": "A00",
"moves": "1. c3 {[%clk 0:03:00]} 1... e6 {[%clk 0:03:00]} 2. Qb3 {[%clk 0:02:57.1]} 2... Ne7 {[%clk 0:02:58.4]} 3. Nf3 {[%clk 0:02:47.7]} 3... d5 {[%clk 0:02:57.9]} 4. e3 {[%clk 0:02:45.6]} 4... Nd7 {[%clk 0:02:57.3]} 5. Be2 {[%clk 0:02:42.3]} 5... a6 {[%clk 0:02:56.9]} 6. O-O {[%clk 0:02:36.6]} 6... b5 {[%clk 0:02:56.3]} 7. d4 {[%clk 0:02:35.1]} 7... Bb7 {[%clk 0:02:55.7]} 8. Qc2 {[%clk 0:02:28.4]} 8... g6 {[%clk 0:02:55.3]} 9. Nbd2 {[%clk 0:02:25.4]} 9... Bg7 {[%clk 0:02:54.6]} 10. Nb3 {[%clk 0:02:23.6]} 10... c6 {[%clk 0:02:54.2]} 11. a3 {[%clk 0:02:21.3]} 11... a5 {[%clk 0:02:52]} 12. a4 {[%clk 0:02:17.9]} 12... bxa4 {[%clk 0:02:49.7]} 13. Rxa4 {[%clk 0:02:16.2]} 13... Nb6 {[%clk 0:02:45.5]} 14. Rxa5 {[%clk 0:02:12.4]} 14... Rxa5 {[%clk 0:02:43.9]} 15. Nxa5 {[%clk 0:02:11.4]} 15... Nc4 {[%clk 0:02:40.4]} 16. Nxb7 {[%clk 0:02:07.8]} 16... Qb6 {[%clk 0:02:36.9]} 17. Bxc4 {[%clk 0:01:59.4]} 17... dxc4 {[%clk 0:02:34.1]} 18. Nd6+ {[%clk 0:01:58.1]} 18... Kd7 {[%clk 0:02:30.1]} 19. Nxc4 {[%clk 0:01:55.2]} 19... Qa6 {[%clk 0:02:28.6]} 20. b4 {[%clk 0:01:49.9]} 20... Ra8 {[%clk 0:02:24.4]} 21. Qd3 {[%clk 0:01:45]} 21... Nd5 {[%clk 0:02:14.7]} 22. Nce5+ {[%clk 0:01:42.2]} 22... Bxe5 {[%clk 0:02:12.2]} 23. Nxe5+ {[%clk 0:01:37.6]} 23... Ke7 {[%clk 0:02:11.1]} 24. Qxa6 {[%clk 0:01:34.8]} 24... Rxa6 {[%clk 0:02:08.5]} 25. c4 {[%clk 0:01:32.9]} 25... Nxb4 {[%clk 0:02:05]} 26. Bd2 {[%clk 0:01:32]} 26... Nc2 {[%clk 0:01:59.5]} 27. h3 {[%clk 0:01:30.3]} 27... Kf6 {[%clk 0:01:55.7]} 28. Rb1 {[%clk 0:01:28.2]} 28... c5 {[%clk 0:01:43.9]} 29. dxc5 {[%clk 0:01:26.3]} 29... Kxe5 {[%clk 0:01:41.1]} 30. Bc3+ {[%clk 0:01:22.3]} 30... Ke4 {[%clk 0:01:39.6]} 31. Rb7 {[%clk 0:01:16.4]} 31... Kd3 {[%clk 0:01:37.9]} 32. Be5 {[%clk 0:01:11.9]} 32... Kxc4 {[%clk 0:01:35.7]} 33. Rxf7 {[%clk 0:01:07.8]} 33... Kxc5 {[%clk 0:01:34.2]} 34. Rxh7 {[%clk 0:01:06.7]} 34... Kd5 {[%clk 0:01:32.4]} 35. Bg3 {[%clk 0:01:03.4]} 35... Ra1+ {[%clk 0:01:28.5]} 36. Kh2 {[%clk 0:00:59.9]} 36... e5 {[%clk 0:01:24.4]} 37. Rh6 {[%clk 0:00:55.1]} 37... g5 {[%clk 0:01:21.8]} 38. Rh5 {[%clk 0:00:52.4]} 38... Ke4 {[%clk 0:01:17.7]} 39. Rxg5 {[%clk 0:00:51.4]} 39... Nb4 {[%clk 0:01:11.7]} 40. Rxe5+ {[%clk 0:00:48.3]} 40... Kd3 {[%clk 0:01:09.9]} 41. Rb5 {[%clk 0:00:44.8]} 41... Kc4 {[%clk 0:01:06.8]} 42. Rb8 {[%clk 0:00:39.9]} 42... Nd3 {[%clk 0:01:02.3]} 43. e4 {[%clk 0:00:35.4]} 43... Kd4 {[%clk 0:00:58.8]} 44. e5 {[%clk 0:00:33.6]} 44... Nxf2 {[%clk 0:00:53]} 45. Bxf2+ {[%clk 0:00:31]} 45... Kxe5 {[%clk 0:00:52.1]} 46. Rd8 {[%clk 0:00:23.4]} 46... Kf5 {[%clk 0:00:51.2]} 47. g4+ {[%clk 0:00:22.2]} 47... Kf4 {[%clk 0:00:49.1]} 48. h4 {[%clk 0:00:20.5]} 48... Kxg4 {[%clk 0:00:47.5]} 49. Rd4+ {[%clk 0:00:19.4]} 49... Kf3 {[%clk 0:00:44.7]} 50. h5 {[%clk 0:00:17.5]} 50... Ra6 {[%clk 0:00:41.5]} 51. h6 {[%clk 0:00:15.4]} 51... Rxh6+ {[%clk 0:00:37.6]} 52. Rh4 {[%clk 0:00:13.9]} 52... Rxh4+ {[%clk 0:00:34.9]} 53. Bxh4 {[%clk 0:00:12.8]} 1/2-1/2"
}
}
Hint
If you have doubts about your regex expressions, check them out on: Regex 101. Also, make sure that your JSON is valid as Regexp may require extra escape slashes.
Slicing¶
You can use slicing to cut values at value[from:to] which is very useful when you are only interested in part of a value. The value is turned into a string with str() before slicing is applied.
String slice¶
Lets say that we have some value like this street-Santas Polar city 45. We would really like to filter away the street- part of that value. And that is exactly what Slicing is for.
{
"name": "root",
"objects": [
{
"name": "fantasy",
"array": true,
"iterators": [{"alias": "data_item", "path": "data"}],
"attributes": [
{
"name": "name",
"data_fetchers": [{"path": ["data_item", 0]}]
},
{
"name": "street",
"data_fetchers": [
{
"path": ["data_item", 1],
"slicing": {
"from": 7
}
}
]
}
]
}
]
}
{
"data": [
["santa", "street-Santas Polar city 45"],
["unicorn", "street-Fluffy St. 40"]
]
}
{
"fantasy": [
{
"name": "santa",
"street": "Santas Polar city 45"
},
{
"name": "unicorn",
"street": "Fluffy St. 40"
}
]
}
Hint
If you have some max length on a database table, then you can use string slicing to make sure the length does not exceed a certain length with the to key. Some databases also has for example two address fields for when the length of one is too short. Then map both with slicing "from" :0, "to": 50 and "from": 50, "to": null respectively and you'll solve the problem.
Slicing numbers and casting¶
You can also slice numbers, bools and any other json value since we cast the value to string first. This means that if you for example get a social security number but is only interested in the date part of it, you can slice it. And then even cast the value to a date.
2020123112345 -> "20201231" -> "2020-12-31"
{
"name": "root",
"objects": [
{
"name": "fantasy",
"array": true,
"iterators": [{"alias": "data_item", "path": "data"}],
"attributes": [
{
"name": "name",
"data_fetchers": [{"path": ["data_item", 0]}]
},
{
"name": "birthday",
"data_fetchers": [
{
"path": ["data_item", 1],
"slicing": {
"from": 0,
"to": 8
}
}
],
"casting": {
"to": "date",
"original_format": "yyyymmdd"
}
}
]
}
]
}
{
"data": [
["santa", 2020123112345],
["unicorn", 1991123012346]
]
}
{
"fantasy": [
{
"name": "santa",
"birthday": "2020-12-31"
},
{
"name": "unicorn",
"birthday": "1991-12-30"
}
]
}
Hint
If you need to take values from end of string, like the 5 last characters, then you can use a negative from value to count from the end instead. This works just like pythons slicing functionality.
If statements¶
Are useful for when you for example get some numbers in your data that are supposed to represent different types.
Simple if statement¶
Let's check if the value equals 1 and output type_one.
{
"name": "root",
"attributes": [
{
"name": "readable_type",
"data_fetchers": [
{
"path": ["type"],
"if_statements": [
{
"condition": "is",
"target": "1",
"then": "type_one"
}
]
}
]
}
]
}
{
"type": "1"
}
{
"readable_type": "type_one"
}
If statements are really useful for changing the values depending on some condition. Check the list of supported conditions.
otherwise can also be used to specify should happen if the condition is false. If otherwise is not provided then output will be the original value.
Chain If Statements¶
if_statements is a list of if statement objects. We designed it like this so that we can chain them. The output of the first one will be the input of the next one.
the DataFetcher object is not the only one that can have if statements, the attribute can also have them. This allows for some interesting combinations.
{
"name": "root",
"attributes": [
{
"name": "readable_type",
"data_fetchers": [
{
"path": ["type"],
"if_statements": [
{
"condition": "is",
"target": "1",
"then": "boring-type"
},
{
"condition": "is",
"target": "2",
"then": "boring-type-two",
"otherwise": "fun-type"
},
{
"condition": "contains",
"target": "fun",
"then": "funky_type"
}
]
}
],
"if_statements": [
{
"condition": "not",
"target": "funky_type",
"then": "junk",
"otherwise": "funk"
}
]
}
]
}
{
"type": "1"
}
{
"readable_type": "funk"
}
{
"type": "2"
}
{
"readable_type": "junk"
}
Using input.json the places that are highlighted is everywhere the value changes.
For input2.json the first if statement is false and no value change. The second if statement is true so value is changed to boring-type-two. The third if statement is false so no value change. The last if statement checks if the value is not funky_type which is true, so the value is changed to junk.
You can even add if statements for every DataFetcher object you add into data_fetchers so this can handle some quite complicated condition with multiple values.
Note
If statements on array and object values behave a bit differently. Have a look in the configuration
Casting values¶
You've learned how to structure your output with objects, find values and asigning them to attributes, combining values and applying if statements. Its now time to learn how to cast values.
Casting values is very useful for when we get string(text) data that should be numbers. Or when you get badly(non-iso) formatted date values that you want to change to ISO dates
Casting is straightforward. You map your value like you would and then add the casting object.
Casting to decimal¶
{
"name": "root",
"attributes": [
{
"name": "my_number",
"data_fetchers": [
{
"path": ["string_number"]
}
],
"casting": {
"to": "decimal"
}
}
]
}
{
"string_number": "123.12"
}
{
"my_number": 123.12
}
Casting to ISO Date¶
When casting to a date we always have to supply the original_format which is the format that the input data is on. without knowing this there would be know way to know fore sure in every case if it was dd.mm.yy or yy.mm.dd
{
"name": "root",
"attributes": [
{
"name": "my_iso_date",
"data_fetchers": [
{
"path": ["yymmdd_date"]
}
],
"casting": {
"to": "date",
"original_format": "yymmdd"
}
}
]
}
{
"yymmdd_date": "101020"
}
{
"my_iso_date": "2010-10-20"
}
Check our the configuration docs on casting for more info
Working with lists¶
Finally! Last topic and the most interesting one!
Usually the data that you are processing is not one thing, but a list of things(data). We want to iterate the list and for each and every piece of data in that list we want to transform it. This is the section that lets you do that.
Lets say we are creating a website for an RPG game that dumps its data in a flat format. Every line is a player with a name, class, money, and x, y coordinates for where he is in the world.
{
"data": {
"something_uninteresting": [1, 2, 3],
"character_data": [
["SuperAwesomeNick", 1, 500],
["OtherAwesomeDude", 2, 300],
["PoorDude", 2, 10]
]
}
}
Now to make the frontend dudes happy we would liketo structure this nicely... something like:
{
"players": [
{
"nickname": "SuperAwesomeNick",
"class": "warrior",
"gold": 500,
},
{
"nickname": "OtherAwesomeDude",
...
}
]
}
Introducing Iterators¶
We can use iterators on an object which works similar to mapping.path, but it applies the current object and all its attribute data_fetchers and nested objects to each and every element in whatever list iterators points to.
Lets solve the above example!
{
"name": "root",
"objects": [
{
"name": "players",
"array": true,
"iterators": [
{
"alias": "character",
"path": ["data", "character_data"],
}
],
"attributes": [
{
"name": "nickname",
"data_fetchers": [
{
"path": ["character", 0]
}
]
},
{
"name": "class",
"data_fetchers": [
{
"path": ["character", 1]
}
],
"if_statements": [
{
"condition": "is",
"target": 1,
"then": "warrior",
"otherwise": "cleric"
}
]
},
{
"name": "gold",
"data_fetchers": [
{
"path": ["character", 2]
}
]
}
]
}
]
}
{
"data": {
"something_uninteresting": [1, 2, 3],
"character_data": [
["SuperAwesomeNick", 1, 500],
["OtherAwesomeDude", 2, 300],
["PoorDude", 2, 10]
]
}
}
{
"players": [
{
"nickname": "SuperAwesomeNick",
"class": "warrior",
"gold": 500
},
{
"nickname": "OtherAwesomeDude",
"class": "cleric",
"gold": 300
},
{
"nickname": "PoorDude",
"class": "cleric",
"gold": 10
}
]
}
iterators is an array of Iterator objects that must contain an alias and a path. alias is will be the name of the key that you will then be able to reference and path is the path to the iterable list/array in the input data.
Note
In our data_fetchers.path we reference the key name(character) which is the alias we set up. Behind the scenes what really happens is that we add this character key to the root input data and run mapping for each val/obj in the list. Its completely fine to name the alias the same as the last key to the in the path. This is demonstrated in the next example. However, it makes more sense to append _item or use the singular form as in list called customers use alias customer.
If you have multiple iterators you must make sure to use unique aliases since otherwise you will overwrite other data.
Iterables Continued¶
So you think that was cool? Well, we can have as many as we want.
consider the following input:
{
"data": [
{
"nested": [
{
"another": [
{"a": "a"},
{"a": "b"}
],
"name": "nested1"
}
]
},
{
"nested": [
{
"another": [
{"a": "c"},
{"a": "d"}
],
"name": "nested2"
}
]
}
]
}
Theres 3 levels of lists. But lets say we want to flatten this structure. Then we will have to get all combinations of data.nested.another.a = a, b, c and d.
Well its easy, first we just add iterators to "data", then we must iterate "nested", then we must iterate "another".
{
"name": "root",
"array": true,
"iterators": [
{
"alias": "data",
"path": ["data"]
},
{
"alias": "nested",
"path": ["data", "nested"]
},
{
"alias": "another",
"path": ["nested", "another"]
}
],
"attributes": [
{
"name": "nested_name",
"data_fetchers": [
{
"path": ["nested", "name"]
}
]
},
{
"name": "value_of_a",
"data_fetchers": [
{
"path": ["another", "a"]
}
]
}
]
}
{
"data": [
{
"nested": [
{
"another": [
{"a": "a"},
{"a": "b"}
],
"name": "nested1"
}
]
},
{
"nested": [
{
"another": [
{"a": "c"},
{"a": "d"}
],
"name": "nested2"
}
]
}
]
}
[
{
"nested_name": "nested1",
"value_of_a": "a"
},
{
"nested_name": "nested1",
"value_of_a": "b"
},
{
"nested_name": "nested2",
"value_of_a": "c"
},
{
"nested_name": "nested2",
"value_of_a": "d"
}
]
And thats it!
Congratulations the introduction course is done!
Time to map some data and have fun doing it!
Also head over to Kaiba Config Generator to do live mapping with a UI
Have a look in usecases section for some quick starts and tutorials