Maintaining data warehouses can be a difficult undertaking for any organization. Not only do you have to establish processes and procedures for regularly loading flowing data, you also have to ensure you’re doing it in a way that’s resistant to failure and future errors. In this post, we’ll take a quick look at some of the biggest challenges of maintaining large scale data warehouses, and how AWS Lambda can help.
ARGGG! The internet went out. What to do while I wait? I know! Write a bash script to let me know when the internet is back up!
Recently I was trying to use the psycopg2 libraries for Python in combination with AWS Lambda. My first hint that this was probably overkill was that the function package, when zipped, started exceeding 50MB. AWS console errors quickly reminded me that 50MB is the size limit for Lambda packages. After correcting a few mistakes (my virtualenv bundled in with my dependencies - Whoops!) and trimming some dependencies I started testing again. This time with a function package just a few bytes under the limit.
My Pluralsight Course is here! After a lengthy Winter vacation of recording and a few weekends of editing I present to you my Pluralsight AWS Developer Course - An Introudction to Lambda