If you’ve implemented any project of relative importance with OL Connect, chances are you had to write scripts at some point, be it in Workflow, the DataMapper or the Designer. In this article, we’ll start looking at ways of improving your JavaScript skillset.

Is JavaScript any good?

Even though the Workflow module allows you to pick from several scripting languages, let’s be realistic: VBScript has been outdated for years and Perl is – to put it politely – quite challenging to master. Python, on the other hand, is a truly powerful language that has tons of extensions and is relatively easy to learn given the slew of online tutorials that are available for it. However, just like Perl, it isn’t natively supported in Windows and it is not suited to client-side scripting. Not to mention that the OL Connect Designer and DataMapper don’t support any of those languages.

That leaves JavaScript.

Full disclosure: I love JavaScript. I love its flexibility. Its versatility. Its documentation. And I love it despite the bad rap it gets from some hard-core application developers: it’s interpreted (so it can’t be efficient); it’s loosely-typed (so it’s a nightmare to debug and maintain); it’s not truly object-oriented (who else uses object prototypes?!?); it requires complex frameworks to accomplish anything (jQuery, React, Angular, etc.).

And you know what? These hard-core developers are not entirely wrong. But they remind me of automobile enthusiasts who insist on using a manual gear stick, turning off traction control and anti-locking brakes, and who never use the backup camera, the lane change assist or the blind-spot monitor.

Now I like cars too. But I’m not nostalgic of the manual choke lever on my Honda 1979. I don’t miss having to stick a screwdriver in the carburetor to keep it open so the car would start by -40 weather. I don’t miss having paper atlases for all of North America stuffed in the door compartments to map out my journey. And I certainly don’t miss my grandpa’s old Citroën model that had – I kid you not! – manual windshield wipers… now who’d miss that?!?

So it’s true that JavaScript suffers from many problems, most of them due to inexperienced developers having built overly-complex solutions and frameworks by bastardizing the language. But that doesn’t mean everyone should have to learn a proper programming language like C++ or Assembly just to be able to massage some data and map it on a page with graphics and fonts. Full disclosure again: I do know several such programming languages. Heck, back in the PlanetPress Suite days, I even contributed to writing an entirely new programming language (PressTalk). But looking back to those olden times, I wish we would just have used JavaScript instead.

But am I any good at JavaScript?

In the world of coding, there is one absolute certainty: however clever, efficient and beautiful your code is, someone, somewhere, can do better. Accordingly, I have long dropped the fantasy of becoming the best coder on the planet. Instead, I strive to improve my skills with every new project I write, regardless of the size of that project.

The first thing to remember when you start coding is that you have to use what you know. Sounds obvious, I know, but it’s important to always start a project with commands, objects and methods you’re already comfortable with. In other words, you can’t improve something that doesn’t exist yet, so you’re better off writing some truly horrible code first and only then – once you’ve established that it works as it should – should you look back at your code and see how it can be improved. This has two benefits: it allows you to deliver your first iteration much faster, and it allows you to defer the improvement to a later time without feeling the pressure of a looming deadline.

Let’s look at a simple example.

Playing with arrays

Say you have an array of values: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34] (yes, math nerds, that’s a Fibonacci sequence), and you want to filter out any value lower than 10. Your first iteration of code might look something like this:

var Fib = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34];
var filteredFib=[];
for(var i=0;i<Fib.length;i++){
   if(Fib[i]<10){
      filteredFib.push(Fib[i]);
   }
} // Result: [0,1,1,2,3,5,8] 

And you know what? That is perfectly good code!!!

How do you know? Well it may not be fancy, yet it does exactly what it’s supposed to do, and it does so efficiently. So that’s your first iteration right there. But is there a way to make this code cleaner (e.g. easier to understand and maintain)? Of course there is!

A quick glance at the JavaScript reference (JavaScript reference – JavaScript | MDN (mozilla.org) – bookmark this right now if you haven’t done so already!) shows that there is a filter() method for arrays. So let’s try that:

var Fib = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34];
var filteredFib = Fib.filter( function(e){ return e<10 } ); // Result: [0,1,1,2,3,5,8] 

Well, well. Dunno about you, but I kind of like this. It’s concise, elegant and efficient. That filter() method is really clever, don’t you think? Well I hate to break it to you, but no, it isn’t that clever. Or to be accurate, it’s no more clever than you are because if you were to take a look at the underlying code for that method, you’d find that it is implemented almost exactly as you implemented it in your first iteration except that it’s slower!

So which iteration of the code should you use? That’s up to you. Some would prefer to use the initial iteration because it is faster (and I mean significantly faster, the more so as your array grows… but more on that later). Others will prefer the second iteration because it is more concise. And others yet – like moi – will look beyond those two iterations before choosing which one should be implemented. But how do you look beyond the code? Easy: you ask questions.

Asking questions

Looking at our two algorithms, it’s obvious that the length of the array is the one determining factor for overall performance. So the first question you should ask is: in the most extreme cases, how big is that array gonna be? If the array can never contain more than a few elements (a few tens or hundreds), then you don’t want to fall prey to the over-optimization monster. Just pick the algorithm you are most comfortable with and move on to your next project.

I can’t stress this enough: don’t over-optimize. I know it might be fun to do, it might lead to some really clever code, you might even win the Nobel prize for algorithmic ingenuity, but really, it’s just useless, and it wastes time. So if you’re thinking of optimizing things that don’t need it… don’t!

A side note about array methods in general, and array.filter() in particular: they are extremely useful in making your code more concise and easier to understand and maintain, but they are generally much slower than if you were to write out the code yourself. The reason isn’t that they are implemented badly by the various JS engines but rather that you are calling an object method and executing a function for each element, which introduces significant overhead in JavaScript’s memory management. So the more elements there are in an array, the more often you incur that overhead, which eventually leads to a noticeable decrease in performance.

That shouldn’t deter you from using those methods (because after all, we don’t often have to deal with arrays containing hundreds of thousands of elements), but it’s something to be aware of nevertheless.

So back to our example, assuming the array could conceivably grow to a significant length, then the second question you should ask is: is the array always sorted? Why is that important? Well because if it’s sorted in ascending order, it means you don’t have to examine every value in the array since you can stop looping as soon as a value is greater than 9. That may not sound like a big optimization, but if your array contains thousands of values, stopping the loop after just a few iterations might have a big impact instead of needlessly looking at values you know are greater than 9.

Assuming the array is always sorted, you can go back to the first algorithm and just add a break statement as soon as a value exceeds 9:

  if(Fib[i]<10){
    filteredFib.push(Fib[i]);
  } else {
    break;
  }

That was easy enough and now we know that, regardless of the length of the Fibonacci sequence that constitutes the array, the loop will always run no more than 7 iterations.

Now can we also implement that optimization in our second algorithm?

No.

The filter() method doesn’t provide a way to break out of a loop, therefore the comparison function is used on every single element in the array. So in this example, now that we’ve asked our second question, we know which algorithm we should be using: the first one.

But what if the array is not always sorted? Then the break statement can’t be used because we still have to examine each array element, don’t we? So we might as well use the more concise, second algorithm. Well… not necessarily. At that stage you should ask your third question: is preserving the array order important?

If it isn’t, then you can pre-sort the array right before the loop and still use the break statement inside it:

var Fib = [0, 34, 1, 2, 8, 13, 3, 5, 21, 1];
Fib.sort(function(a,b) {return a-b});
for(var i=0;i<Fib.length;i++){
   ...
}

Don’t worry if you are not familiar with the syntax for the Sort() method, just read up on it through the link provided above. What you do need to remember, though, is that sorting an array comes at a cost: pre-sorting the array before filtering is likely to take significantly more time than filtering an unsorted array. At this point, I should also point out that JavaScript’s native sort() method is not the most efficient way of sorting elements. Implementing your own QuickSort algorithm is much more efficient… but wouldn’t that fall under the over-optimizing category?

In any case, if preserving the array order is important, then you cannot pre-sort it anyway so you might as well use the second algorithm.

Oh, by the way…

Just as you turn in your project and get ready to pop open a can of your favorite drink to celebrate a job well done, your boss tells you that your code doesn’t work because the resulting array contains duplicate values and he swears he made it crystal clear to you that those were not allowed.

*sigh* … we’ve all been there…

You look back at your algorithm and try to figure out how to best do this. You have two options: either eliminate duplicates from the source array or from the result. Either way, you’re going to have to loop through one array and remove duplicates. Which one? There’s no definitive answer here and I personally think it doesn’t make much of a difference.

But how do you eliminate duplicates? One way to eliminate them is to store values in another array, checking each time if that other array already contains the value:

var Fib = [0, 34, 1, 2, 8, 13, 3, 5, 21, 1];
var noDuplicates = []
Fib.sort(function(a,b) {return a-b})
for(var i=0;i<Fib.length;i++){
  if(Fib[i]<10){
    if(noDuplicates.indexOf(Fib[i])==-1){ // Check if value already exists in resulting array
      noDuplicates.push(Fib[i]);
    }
  } else {
    break;
  }
} // Result: [0,1,2,3,5,8,13,21,34] 

Now again, that is perfectly valid code. But with each iteration, you are looking up in the resulting array to see if the value already exists in there. And as that resulting array grows, each lookup becomes increasingly expensive in terms of time. That’s because the indexOf() method internally performs a loop on the target array until if finds – or not – a specific value. If the number of unique values is in the thousands, this method will have a negative impact on performance.

Surely, there must be a more efficient way of doing this. So you have to ask yet another question, but this time you’ll take it to Google: how to remove duplicates from JavaScript array? You will get millions of hits, but after reading just one or two of them you’ll find the answer: Sets.

JavaScript conveniently implemented Sets, which are collections of unique items. A Set is not an array, but it is array-like. One of the great things about Sets is that you don’t have to check if they already contain a value before adding it. You just add it and the Set does all the magic for you. So the solution is to take your source array, store it in a Set (which will automatically remove duplicates) and then convert the Set back to an array that can then be filtered:

var Fib = [0, 34, 1, 2, 8, 13, 3, 5, 21, 1];
var newSet = new Set(Fib);
var noDuplicates = Array.from(newSet).filter(function(elem){ return elem<10 } );
noDuplicates.sort(function(a,b) {return a-b});

// Result: [0,1,2,3,5,8,13,21,34] 

And if you really want to go nuts and write the entire thing with a single statement, you can take advantage of method chaining, which allows you to use successive methods on the results of the previous one:

var Fib = [0, 34, 1, 2, 8, 13, 3, 5, 21, 1];
var noDuplicates = Array.from(new Set(Fib)).sort(function(a,b){return a-b}).filter(function(e){return e<10});

But with that kind of coding, you are at risk of slowly turning into one of those hard-core coders I mentioned at the top of this article. So let me ask you this… do you drive stick-shift and shun that useless back-up camera on your car? (One last full disclosure: I do…)

Conclusion

In this article, I wanted to demonstrate that any project that uses JavaScript can be implemented in a variety of ways, but that you should always start by using the methods and commands you’re comfortable with. More importantly, I wanted to show how to approach – or avoid – the optimization process by asking the right questions.

In a future article, I will tell you how I became more proficient with JavaScript, after years of coding with other languages.