Process files and move them when processed
I use AWS SES for incoming emails. It drops text files into an s3 bucket and then I need to parse these into json documents. There did not seem to be any means of setting this up within data-lake. I also need to move documents from the inbox folder to the processed folder once they have been added to the database. Was hoping data lake could manage all my ingest etl for me and all I need to do is worry about talking to the database once set up, at this point I think I will just implement the above in lambda and push docs into my database and just skip the data-lake functionality altogether.
7
votes
Andrew
shared this idea
-
Charlie
commented
Could we create an example of using AWS Glue to move raw data into catalogued S3 buckets then queried with ADL?