Montag, 24. Oktober 2016

A mapping did not load any data containing an SQL override in a source qualifier using a mapping variable as filter

First, I know the setup is cheap. This project has the segregation of environments for DEV, INT and TEST made by differently named workflows in the same folder of the same repository... and we use mapping variables to store the highest ID having been loaded and to detect the records still to be loaded. Not obvious to me, the value of the mapping variable stored the repository is not saved per mapping but at most per folder maybe even per repository. I did not test. Being aware of that, it explains the problem described in title. The "failing" mapping did not load anything because it is the INT mapping using the stored value of the variable stored by DEV/TEST. So, if the source is the same (and it is) you have loaded everything already in DEV/TEST and the detection of records to load fails miserably.

Things learned

  • Do not be too cheap
  • Do not use mapping variables to detect whether rows have been loaded already or not. (This is not necessarily restartable anyway.)

Donnerstag, 20. Oktober 2016

Rank Transformation is spezial filter (I want the top BLUPP of the groups BLÖRTZ of the data)

A colleague of mine asked me about the working of the rank and I had to admit I did not know much about it. I just did not use it and might get using it more often instead of some more or less complicated sort, aggregate and filter conjunction. However, we tried it out a bit. Here our findings
  • It is not just a ranking thing but it almost always filters!!! Number of Ranks property controls this. From each defined group, it returns as many rows as it is defined here. Again, rows not ranks as the name suggests. I.e. if you set it to 1 (well, this is the default) and in one group there are several records with rank 1, you will still get only one of those, probably just the first one happening to get identified as rank 1. To, hopefully, switch off filtering, set the number of ranks to its maximum value (for version 9.1: 2147483647)
  • You control the grouping by the ticks in the Group by column of the ports tab. You can group over several ports.
  • Obviously, you need to have a sort criterion. This is controlled by the R column of the ports tab. Surprisingly, you can rank over one port only. That is, if you need to sort over several ports, you'll have to compound/concatenate them beforehand into a sorting/ranking port.
  • The Top/Bottom property tells the sorting order, i. e. Top means descending, Bottom means ascending.