Lucy dreaming up the next big adventure!

The straight ahead way of using AWS step functions is to daisy chain lambda after lambda to accomplish a task.

Naturally, apps and workflows can get more complex. There can be a need to limit the number of concurrent tasks. Or, inversely, the need to run massively concurrent tasks at once and then resolve them.

Below are a few tools to orchestrate workflows with added complexity.

SQS

AWS Simple Queue Service is just that: A message queueing system that can hold messages for you and send off data when resources are made available. You can tie a lambda function to an SQS as a trigger. In that case, when a message comes through and is ready for consumption, the lambda function can take the message and do work from there.

An important piece here is that you can limit the number of concurrent functions at any given time while also holding on to impending tasks. If you need to limit a task to 100 concurrent calls, this is where you can do it.

Distributed Map

One possible state type in Step Functions is "Map". Map allows your step function to take a piece of data and iterate over it with an assigned process.

Previously, inline maps had a limit of 40 methods at a time. For a much more powerful implementation, switching to the distributed mode allows for up to 10,000 parallel executions.

The key role a distributed map step will play is in allowing for the whole computational process to still be contained within the step function.

Since I introduced the SQS above, one possible solution is simply to pop my list of data into an SQS without the concurrent limit. However, if you need to finalize and process the result from those individual computations, coordinating their completion and status manually can become complex.

With Distributed Maps, you still have the benefit of massive concurrent computation, while also having the benefits of efficient resolution and simple orchestration.

Callback with Task Tokens

While distributed maps allow controlled parallelism of a dynamic number of functions, introducing SQS adds an extra layer of variability. We don't know when our message sent to SQS will be fired and concluded.

Say I send a message to SQS in the middle of my step function and am dependent on the returned results. How can I instruct my step function to wait for those results?

Callbacks with Task Tokens are the answer. The idea is that by setting our state resource to arn:aws:states:::sqs:sendMessage.waitForTaskToken, we can instruct our step function to pause here. When sending a task token to SQS along with our payload, on completion, we can send a success or failure message back to our step function. When it comes with the original task token, AWS knows to then continue the initial step function with the returned results.

Putting It Together

With all of the above, you can handily stitch together a workflow that:

• fires of concurrent tasks through a Distributed Map
• Siphon off select tasks to an SQS for a separate, limited process.
• Return the SQS results to the original step function with a task token.

Listen on Youtube

And I'll always see...Polka Dots and Moonbeams

When I kiss that pug-nosed dream

The scene I wake up to. 💛

Listen on Youtube

Getting back in the saddle 🎷

🍕🌵

Listen on Youtube

Skylark~

Have you anything to say to me?

Won't you tell me where my love can be?

⛰️

Listen on Youtube

Recording continues! A really short snippet this time — because it took all week to get this lick up to speed. 😂

More scenes from our twilight walks at home.

I spent a decade in creativity with clear goals and a seemingly direct path for success. Get the degrees, play with the right people, practice the right things, serve other people along the way, and a fulfilling career in music awaits on the other side!

Now, though, there is no clear path. There's no career to pursue, no obvious prize to chase, and there aren't all encompassing models for the things I want to make.

While I learned a great deal from this more success driven phase of my life, met wonderful people, and grew plenty as a person, I'm now looking ahead at new terrain.

What do you do when that script runs out? What do you do when you reach the end of where that map leads?

. . .

Earlier this week Miranda and I watched Rocky, both our first time seeing the classic film.

A hobbyist boxer with natural talent, the film finds him continually training and taking on small fights. A once-in-a-lifetime opportunity comes with a chance to fight for the championship against Apollo Creed.

To jump to the most striking moment in the film for me: Rocky finds himself nervous the night before the fight. He doesn't know that he can win. But, he commits to "going the distance," to staying in the ring through all 15 rounds. The refs ultimately call Apollo the winner, but only after 15 tough rounds, Rocky is still standing at the end.

These days, that's where I find myself creatively.

I've had to relearn enjoying the practice for it's own sake. After spending time with contests to perform in and hoops to jump through, it's easy to get caught up thinking there's something the work needs to "win" with.

Anytime I find myself using the old map for my practice, a sense of defeat comes in and motivation is zapped. Without knowing it, a sublte voice has been asking "What is this going to get me?" And it's been the wrong question.

The better question: "How can I keep doing this for as long as possible?" "This" being any medium and practice that sparks imagination and stirs something deep within. At present: music, art, writing, and software.

With Rocky, the better aim is staying in the ring. How can we roll with the punches, get back up when we're down, and continue engaging round after round?

Later films, naturally, follow a journey where Rocky does eventually win it all. While it's a satisfying story when the conclusion puts our hero on top, with all of the accolades and success in the world, I can't help but think the first movie in isolation has a more meaningful message. Regardless of the outcome, victory comes from staying in until the end.

Listen on Youtube

Working on recording something new! 🎶

Dreaming of colder weather. ⛄️

I've been between projects in Python, Java, and JavaScript. Aside from the syntax differences, I've been pausing to wonder why the context switch takes time.

The answer has to do largely with the programming paradigm these languages tackle programming challenges.

There's a spectrum here between Functional Programming and Object Oriented Programming. While all three can manage both paradigms in this day and age, their communities favor being on different poiints along the spectrum:

• JavaScript leaning heavily towards functional programming
• Java towards Object Oriented Programming
• Python sitting in the middle with great support for both paradigms

If you've started in web development, more than likely functional programming is very familiar. So here I'll explore how to make the switch in thinking in an OOP way coming from that perspective.

Functional Programming

At the risk of stating the obvious: FP is built around functions. These individual functions are not tied to an object. They take inputs, process those inputs, and return a result.

const add_then_multiply = (a, b, c) => {
    let res = a;
    res += b;
    res *= c;
    return res;
}

What my silly example demonstrates is how a function takes in a handful of arguments, calculates through them, and returns the result. Side effects are possible, but not desirable. This paradigm fits nicely with unit testing and TDD because it's simple to test. If I put in 1, 2, 3, I expect to get 9.

JavaScript takes this a step further by treating functions as first class citizens. A function can be passed into another function (referred to as a callback) and then call that function within its own procedural process.

To oversimplify the pros — we are namely looking at a process that thrives on specific calculations and processes. This can lend short programs to be easy to read and unambiguous. When avoiding side effects, there's no confusion around what the program is doing.

Switching to Object Oriented Programming

Functions exist in OOP, but the major difference is that they are tied to data. I'll sum up my case right here: If you're looking to switch your thinking between the two, start thinking about how groups of data need to operate and can interact with each other. This is different from thinking of only the functionality that needs to occur.

In Python, say that I have a representation of a web page. I want to encapsulate the data and some functionality around it in a class:

class WebPage():
    def __init__(url: str, name: str):
        self.url = url
        self.name = name
        self.visits = 0
        
    def visit_url(self):
        # Open url
        
    def print_webpage_details(self):
        print(f'{self.name} at {self.url}')
        
    def add_visit(self):
        self.visits += 1

This simple example demonstrates the shift. Not only have I defined some functionality, but that functionality is primarily centered around manipulating and using the data that's present on the instance.

Imagine it multiplied by dozens of classes then. All with their own state and functionality.

In fact, if you've been writing React, you've likely already been thinking in a moderately OOP way! The execution is largely functional. Hooks are primarily event driven. But it's the encapsulation with state and methods that also blends a bit of OOP into the way React is written.

There's more complexity that can naturally be held this way. And the power comes from having some flexibility to employ both. A procedural script employing functional programming that leans on Objects to encapsulate large amounts of data and complexity is what most apps realistically employ. So it's not a matter of one or the other, but understanding what problem you're trying to solve and picking the right paradigm for the job.

Listen on Youtube

My pup and I missed each other while I was in Chicago this week!