Spark Up What You Have

Ahad L. Amdani • May 7, 2016

The Series – Table of Contents

An Overview

Introduction to Spark
Spark Up What You Have

The Backend

The Spark Philosophy
A Closer Look at the Innards of Spark

Making Spark Your Own

Contents-at-a-Glance

  1. Introductory (Warning) Words
  2. “Should I integrate Spark?”
  3. Backup your database, then DROP your tables
  4. Start with Spark, then integrate what you have
  5. Alter and “refresh” your migrations
  6. Modify your configuration/.env file
  7. Refresh your dependencies
  8. Face your fears: run your migrations!
  9. Restore your Data or ETL off of your backup

Introductory (Warning) Words

If you've read the previous article in this series, you've probably already realized what Spark is and what it has to offer, but to sum it up quickly: Spark is a boilerplate to bring a set of features out of the box for a new (typically SaaS) application or idea:

But, heed the warning: Spark is intentionally designed for new applications. It is not built for integration, nor is it friendly or easy to integrate Spark into an existing application. But it is possible. It's just a bit of a headache, and you'll have to backup your data and drop your tables. Sounds scary, right? Well, it is, and now you can't say you haven't been warned. Because you have been, you know? Just now. I'm warning you. It's dangerous.

"Should I integrate Spark?"

Well, there are plenty of use cases for why you would want to integrate Spark. I won't list them all, but it can be well worth it to absorb the hit of integration so you can gain some or all of the features provided by Spark that you don't want to or have to develop yourself.

Have a great app already? Just collecting information or data before adding in some of the features above that you could really use, but didn't want to spend the time to build out? These are great use cases. They may seem to be contrived, but you'd be surprised how many people this is the exact case for, and it's clearly non-trivial, since I've warned you multiple times that it Isn't for the faint of heart.

One of the many other reasons you may want to bite the bullet and integrate Spark is because it's a boilerplate that will be receiving updates and upgrades. Updates will include bug fixes and possible tips and tricks, as well as user experience enhancements (1.0 -> 1.x). And then there are upgrades (1.x -> 2.0). Sure, there will be a cost to upgrade – but there will also be new, additional features. If you take the time and effort to integrate now, when it's difficult, but not ridiculous to integrate, then when you upgrade it to the next iteration, you could get a lot of these new features for (relatively) free, and that can probably save you hundreds of hours and tens of thousands of dollars of developer time.

Then again, Spark 2.0 may not be a straightforward upgrade for 1.x users. It may, in fact, also be built and designed to be used from the ground up or for a fresh app install, so just be cognizant of that possibility.

But hey, that'll give me an opportunity to write another guide! Um..you're welcome?

Backup your database, then DROP your tables

Yes. That was a shiver that ran down your spine. But it's absolutely important that you DROP your database tables to make room for the "fresh" migrations that will be generated by the Spark installer. Of course, BACKUP your database – schema & structure, data, everything. Hell, clone the damned thing and put it up in an (encrypted) cloud!

Of course, I expect (and implore!) you to read this entire article and understand the theory in its entirety before you rush off to do anything. Twice.

I also expect you to be doing this in your development environment first, not your staging or production environments. You do have a dev/stage/prod setup, don't you? I mean, you can setup continuous integration as well as continuous deployment all the way through a swappable staging server, and it really Isn't that hard (once you get used to some concepts and tools). Maybe I'll cover that in a different post.

However, my side note here still stands. Do this in development first. Make sure things work as you expect them to. Then, once you're ready to move things into production, stop. Schedule a time when the application has the least amount of traffic/impact with respect to a service interruption. Bring everything down. Back everything up. Thrice.

Then, without dropping the database or the role/account that accesses the database, very specifically and carefully, drop your production database tables whilst asking for forgiveness from on high.

Start with Spark, then integrate what you have

Let me make this emphatically clear: Spark does not (out of the box) integrate with your existing application. It will overwrite your routes file, delete your migrations and insert its own, and mess with your project structure. So, how do you take your app and put Spark in it? The answer is: You Don't! You create a brand new Spark installation with the same name as your app, test that it's a clean install and that everything works, and then integrate your application on top of Spark.

This is not a joke. This is the absolute easiest way to do this, and provides you with the most amount of automation (that is to say, some automation) to minimize the task at hand (by a minor amount).

Using Git?

If you're using git and want to track your progress, create a new feature branch in your existing application's git repo, then delete everything besides your .git folder, .gitignore, and your configuration/.env file. Commit this "clean" state before beginning the Spark installation above. Then, once you've got a working copy of Spark and all of its features, make another commit. In this manner, you've got a clear history of the previous app and all of the steps you've taken to get the application working with Spark, and can always roll back to this commit as needed to work out any kinks that will occur over the next few steps.

And there will be kinks. But that's okay. You'll iron them out in your handy, dandy development environment, completely away from your staging and production environments, and you'll make plenty of backups. Right?

Trust me: source control and backups. It's like sunscreen from that graduation speech song. You don't want to be pulling your hair out like I did when I screwed this up multiple times. And you definitely want to make commits on a per-step basis so you have a rollback available at any given point. I would also make a copy or backup of your database on a per-step basis.

All the rest is just my meandering advice. But *trust me* on the sunscreen.

So what does this mean for you? Follow the steps in the Spark Installation Documentation to create a clean Spark installation that's got up to date dependencies from Composer and through Node, and compiles all of your assets through Gulp. Then, run your Spark migrations. Once you've generated your views and confirmed you've got a clean Spark installation, you're ready to integrate your existing application.

Alter and "refresh" your migrations

Now, while this part is by far the most important, it can get a bit tedious. You're going to perform a few passes over two distinct steps here. One, you're going to analyze every table/migration that's been provided to you by Spark to identify the overlaps in your database's existing schema/structure (as well as your existing application's model files). Then, taking care to make the appropriate "add" and "remove" migrations that affect those overlap tables (such as the users table if your existing app made any modifications to the columns in it), you're going to ensure that the overlap tables retain every column and field needed to make Spark work while also bringing in any fields needed to make your existing application function. That's step one.

Step two is to "refresh" your migrations for the non-overlap tables. You're going to basically want to copy and paste the content of these non-overlap migrations in newly generated migrations (and thus, titled with a later date stamp than the Spark migrations which were generated by the installer).

So, repeat the steps after me: (1) alter the overlap migrations, using the existing application's original migrations as a base, to work with Spark, and (2) refresh the non-overlap migrations from the existing application.

As a reference, here are the original migrations. I've bolded the ones you'll definitely want to see if there are any overlaps for in case you have something similar in your existing application. More than likely the non-bolded migrations shouldn't have any issues, but your mileage may vary. The migrations are:

  1. Announcements
  2. API Tokens
  3. Invitations
  4. Invoices
  5. Notifications
  6. Password Resets
  7. Performance Indicators
  8. Subscriptions
  9. Teams
  10. Teams/Users
  11. Users

Modify your configuration/.env file

You want to absolutely make sure you’re still using your old app’s Application Key (APP_KEY) to ensure password hashing will still work. You should also make sure that you’ve essentially copied over all of your previous environment or configuration files’ keys.

You also have to make sure you retain the keys needed by Spark. You should be able to just retain either the Stripe OR the Braintree. All keys are as follows:

  1. AUTHY_SECRET
  2. STRIPE_MODEL
  3. STRIPE_KEY
  4. STRIPE_SECRET
  5. BRAINTREE_MODEL
  6. BRAINTREE_ENV
  7. BRAINTREE_MERCHANT_ID
  8. BRAINTREE_KEY
  9. BRAINTREE_SECRET

Refresh your dependencies

This seems a bit obvious, but after having done everything else, you can never be too sure. This did bite me a couple of times, so I listed it as an explicit step just so we're sure we do it: Don't forget to composer update/install and npm install. Oh, and don't forget to re-run gulp either if you don't have watchers setup.

It's easy to list these steps as if it's something you can do in a few minutes or even hours. Sometimes this stuff takes days and dependencies get updated. Run these commands. If everything is still in a good state (despite not having run your migrations yet), make a commit, or at least stash the changes once.

Also, to help: print the article out and cross out the steps as you encounter them. It may keep you sane, but if it doesn't, blame Google. It's all Google's fault.

Face your fears: run your migrations!

Now comes the moment of truth: run your migrations! Assuming you've done everything correctly from a syntax and a relationship/structure point of view, you'll get output indicating that all of your altered migrations took, and of course your refreshed migrations will also be completed.

The important part here, despite a successful completion of your migrations, is to test to make sure your previous application structure works on top of and in addition to your newfound Spark boilerplate and feature set. In the case of column (or table) names you've changed, or settings that may have been slightly altered in order to accommodate Spark, you'll want to go back through your application code and update those references. Depending on the level of changes, you may even have to massage the structure you had previously into a shape that interacts well with Spark, but that will be unique on a per application basis.

Restore your Data or ETL off of your backup

This is an interesting exercise because your action will depend on how much you've had to modify your data. Theoretically, you should just be able to restore most of your data into the refresh tables from the backup you made at the beginning of the process, including even the overlap tables (with some null values for the new columns inserted by Spark, or some reasonable defaults). However, in case there were significant changes, as well as if you had to alter some table names or alter several columns because of the Spark migrations, you'll need to setup some ETL (Extract, Transform & Load) processes to take the original source data, manipulate it to what the new tables/columns are expecting, and upload it in place.

There are some great tools out there to complete this step, depending on your database. I would highly recommend Navicat for PostgreSQL and MySQL databases, or the 3T products for Mongo databases. SQL Server Management Studio offers everything you need to handle things in case you're working with SQL Server.

Ideally, once you've restored all of your data, your integration of Spark and your existing application is complete, and you should be able to walk through the entirety of your existing application, and review that the proper values appear in your database-driven dropdowns, configuration settings, etc. Once you've confirmed this is the case, commit and backup your newly integrated application's database so you have another rollback step (just-in-case). Congratulations! You've successfully integrated the generic boilerplate and feature set into your application! And we can always ignore the fact that it was actually a reverse integration :)

Hi, I'm Ahad L. Amdani and I'm a craftsman of code and I build solutions to help make a difference. if there's one thing you should know about me, it's that I'm passionate about business and technology. I like to create things that generate business value and most of the time, I use technology to do that.

You can find out more about me and follow me with the following links:
Website: ahadamdani.com
LinkedIn profile: linkedin.com/in/ahadamdani/ Twitter profile: @ahadamdani
GitHub profile: .git/aamdani
Medium: @ahad

I'm currently working with etc.io and my consulting business, Methodology, LLC.