One question I keep getting from new Magento 2 developers, (and we’re all new Magento 2 developers) is, “How should I organize my project files?”. Since Magento has heavily restructured code organization around Composer, it’s not always clear how files should be organized during development, organized for distribution, and how (if at all) to move between these two modes.

While this article won’t definitively answer those questions, we will dive into some of the formalization behind Magento 2 components, as well as how Magento 2’s Composer integration works. With this information in hand, you should be able to come up with a project structure that works for you and your team.

Magento 2 Components

Magento 1 had an informal, inconsistent idea of components. The best way to think about Magento components is

A group of source files, in various formats, with a single purpose in the system

If that’s a little vague, see our previous comments about informal and inconsistent. Speaking more concretely, the four component types in Magento are

Modules

Themes

Language Packs

Code Libraries

In Magento 1, modules were pretty well defined and self contained

A folder of files with an etc/config.xml, with developers making Magento aware of the module via an app/etc/modules/Package_Namespace.xml file

Themes were a little less defined and a little less self contained

A collection of files under app/design/[area]/[package]/[name], with developers making Magento aware of the theme via a setting in core_config_data. Unless it’s the admin theme in which case you need a module. Also, modules are responsible for adding the layout XML files to themes. Also, while we’re here, modules aren’t all that self contained either because if they want to use phtml templates then the templates need to be in a theme folder

Things start to get really vague with language packs

A collection of key/value csv files located inapp/locale/[language]/Packagename_Modulename.csv. Also we’re just going to drop email templates in here because reasons

And Magento 1 barely had the concept of a generic code library.

Um, yeah, maybe just drop them in lib and add that path to PHP’s autoloader? And look in the code pool folders too? And maybe just add a top level js folder for javascript libraries? Unless they go in skin?

Oh right! Skins! Magento 1 also had a (now dropped) concept of skins. Skins were best defined as

Any CSS or javascript file that doesn’t belong in a theme.

Where CSS or javascript file that doesn’t belong in a theme was defined as

Any CSS or javascript file that doesn’t belong in a skin

While, in practice, norms developed over time and development wasn’t as chaotic as I’m describing, Magento 1’s lack of formalization around components did make the system harder to work with, particularly if you were trying to redistribute Magento 1 code for reuse.

Magento 2 formalizes the idea of components, and this formalization means the core Magento system, and other external systems (i.e. Composer) can deal with these components in a sane and reasonable way.

Magento 2 Components

In Magento 2, a component is

A group of files, under a top level directory (with sub-folders allowed), with aregistration.php file defining the type of component.

That’s it. There’s nothing about how the components work, interact with the system, or interact with other components. Those things aren’t the concern of the component system.

As of this writing, there are four component types in Magento 2

Modules

Themes

Libraries

Language Packs

Let’s take a look at this in action. Consider the Magento_AdminNotification module. This is a collection of files, under a top level directory (AdminNotification), with aregistration.php file. If we take a look at registration.php

We can see this file registers a component via the static\Magento\Framework\Component\ComponentRegistrar::register method. This component is a module (\Magento\Framework\Component\ComponentRegistrar::MODULE), its identifier isMagento_AdminNotification, and you can find its files in the __DIR__ folder, (i.e. the same directory registration.php is in via PHP’s magic __DIR__ constant).

Next, consider the Luma theme. Again, a collection of files, under a top level directory (luma), with a registration.php file.

Here the component type is a theme (\Magento\Framework\Component\ComponentRegistrar::THEME), and its name isfrontend/Magento/luma.

Even though the Magento GitHub project has these files in familiar locations, (app/code,app/design, etc.), thanks to Magento 2’s new component system, these directories can be located anywhere, so long as as the module, theme, library, or language pack correctly defines its registration.php file.

How Magento Loads Components

At this point, the systems minded among you are probably wondering how Magento 2 loads and identifies components. It’s one thing to say so long as the module, theme, library, or language pack correctly defines its registration.php file, but the system still needs to load these files, and that means there are rules.

In order to get Magento to recognize your module, theme, code library, or language pack (i.e. your component), you need Magento to read your component’s registration.phpfile. There are two ways to get Magento to read your registration.php file

Place your component in one of several predefined folders

Distribute your module via Composer, and use Composer’s autoloader features

Of the two methods, the second is the preferred and recommended way of distributing Magento 2 modules. For development, the first offers a convenient way to get started on a component, or checkout/clone a version control repository to a specific location. The first also offers a non-composer way for extension developers to distribute their components.

Predefined Folders

At the time of this writing, Magento 2 will scan the following folders/files for components (the patterns below are for the glob function).

This is what allows you to place modules in app/code/Packagename/Modulename, or themes in app/design/[area]/[package]/[name], etc. Magento will explicitly look forregistration.php files to load at these locations. Also of interest are theapp/code/*/*/cli_commands.php files — this appears to be a way for a module to register command line classes without using di.xml.

It’s not clear if these folders were added as a stop-gap measure while Magento 2 gets everyone moved over to Composer distribution, or if they’ll stick around for the long term. If you’re curious, Magento does this registration check in the following file

If you’re researching how a future version of Magento 2 handles scanning for components, this would be a good place to start.

Composer Distribution

The other way to have Magento notice your component is to distribute your component via Composer. We’re going to assume you have a basic familiarity with Composer, but for the purposes of this article, all you really need to know is

Composer allows you to ask for a package of PHP files, and have that package downloaded to the vendor/ folder

So, assuming you have your Magento component in GitHub (or a different source repository Composer can point at), and your component has a registration.php file, the only question left is How do we get Magento to look at our registration.php file.

Rather than have Magento scan all of vendor/ for registration.php files, (an approach that could quickly get “O^N out of hand” as the number packages grows), Magento uses Composer’s file autoloader feature to load each individual component’sregistration.php file.

If that didn’t make sense, an example should clear things up. Assuming you’ve installed Magento via the Composer meta-package, or installed it via the archive available via magento.com (which is based on the meta-package), take a look at the catalog module’scomposer.json file.

The files autoloader section was originally intended as a stop gap measure for older PHP packages that had not moved to a PSR-0 (and later, PSR-4) autoloader system. Composer’s autoloader (not during install or update, but when your application is running) will automatically include any files listed in files (with the specific package as the base directory), and package developers can do whatever they need to do to setup their pre-PSR autoloaders.

Over the years, many frameworks have taken the simplicity and flexibility of the filesautoloader and turned it to different purposes. Magento 2 is no exception. The above autoload configuration ensures Composer will always load the file at

This, as we’ve already learned, will register the component. The same holds true for third party components (i.e. yours!) — make sure you’ve created a registration.php file with the correct registration code for your component type, and then include an identical filesautoloader.

Invalid Assumptions

There’s one last important thing to take away from this, even if you’re not responsible for packaging your company’s Magento work. It’s no longer safe to make assumptions aboutwhere a folder is located in located in the Magento hierarchy. If you’re trying to find a specific file in Magento, it’s more important than ever to learn your way around theMagento\Framework\Module\Dir helper class.

Daily Work

So, now that we have a better understanding of what a component is, and how Magento loads components into the system, that still leaves us with our original question. Where should the code for our in progress Magento projects go? How should we store our projects in source control?

Unfortunately — there’s no clear answer, and a lot will depend on the sort of project you’re working on. Are you an extension developer? A theme developer? A system integrator/store builder or someone integrating with a Magento system? Do you want your working source repository to be the same repository Composer reads from? What tooling is your team is familiar with? While there certainly are approaches that are “better” for each scenario, from a programmer’s point of view Magento 2’s still too new to know for sure.

For what its worth, I’ve been creating symlinks to my source repositories so thatNonComposerComponentRegistration.php finds my components, using a build process to create the final Composer accessible repository, and temporarily patching over any issues Magento has with symlinks.

Part of being a Magento 2 developer will be figuring this out for your own team, even if you’re just a team of one.

In our last article, we talked a bit about Magento 2’s use of Composer, and touched on the Composer meta-package installation method. One high level take away was, when you use Composer’s create-project method to start a new Magento project, you are

Fetching the latest version of a specific Composer package (magento/project-community-edition) hosted at repo.magento.com

Running composer install for that project’s composer.json file

The curious among you may have noticed something strange about this. If we use the --no-install option to skip step #2 above (i.e. only fetch magento/project-community-edition),

The README.md file is a basic Welcome to Magento affair, and the composer.json file contains the actual Composer packages you’ll need to install Magento. That’s whymagento/project-community-edition is called a “meta” package — it’s not he actual package, and most software engineers aren’t english or philosophy majors and enjoy stretching the definition of “meta”.

The update folder contains a snapshot of the Magento Component Manager application (i.e Market Place updater), which is a separate project from the Magento core. Why this feature is a separate application, and why Magento distributes it like this is a story for another time.

All in all, relatively straight forward. However, after running composer install (or running create-projectwithout the --no-install flag), we end up with the following

That’s a huge difference. A plethora of files and folders. Many of these files are necessary for Magento to run, others are informational, and still others are sample configurations. However, if you’re new to the Composer eco-system, you may be wondering

Where the heck did all these extra files and folders come from? I thought Composer would only update files in /vendor.

Today we’re going to explain how these files get here, and why you’ll need to be hyper aware of this as a Magento 2 developer. To start, we’ll need to dive into some less known features of Composer

Composer: Plugins and Scripts

Composer bills itself as a Dependency Manager for PHP. While this is true, and dependency management is an important part of a PHP project, Composer is really a foundational framework for PHP development, and serves the same role that linkers do in the C/C++ world.

Yes yes, I know, from a computer science point of view linkers and Composer couldn’t be further apart. However, the end result of a linker is, the C programmer stops needing to worry about how they incorporate code from other libraries into their program. In a similar way, Composer does the same thing for PHP — if a project conforms to what Composer expects in terms of directory structure and autoloading, and a PHP developer conforms to what Composer expects from a PHP program (i.e., includes the Composer autoloader), the developer stops needing to worry about how they should include other people’s code in their own systems.

When considered from this point of view — that Composer is, itself, just another programmatic framework that your code sits on top of — it makes more sense that Composer would have a plugin system for changing, altering, and extending its behavior. There are two main systems programmers have for altering the behavior of Composer. These systems are scripts, and plugins.

Scripts and plugins share a base set of concepts, but have a few key distinctions. Scripts provide a way, in the projectcomposer.json file, to take additional programmatic action when composer triggers certain events. These events are listed in the Composer manual, and include things like pre-and-post composer install running.

Plugins, on the other hand, provide the same mechanism for individual packages that are part of a larger project. In addition to listening for Composer events, plugins also have the ability to modify composer’s installation behavior.

Put another way, you configure scripts in your main composer.json file, you (or third parties) configure plugins in composer.json files that live in the vendor/ folder.

While understanding both systems is important for a well rounded Composer developer, today we’re going to focus on the plugin system.

Plugin Example

Rather than try to describe things from scratch, we’ve created a simple Composer pluginthat should demonstrate the plugin lifecycle, and help you understand what Magento 2 is doing with Composer plugins.

you can see that this plugin listens for the post-install-cmd and post-update-cmdevents. You tell a plugin which events it should listen to by defining agetSubscribedEvent method that returns an array in the above format. Keys are the event, and values are the method, (on the plugin class), that Composer calls as an observer.

In our case, both the post install and post update events call the installOrUpdate method, and this method logs some simple information to the /tmp/composer.log file in our temp directory.

Composer calls the activate method when it detects the plugin every time Composer runs. The activate method is where you instantiate any other objects your plugin will need. In our case, we’ve added a line to log when the method is called.

All in all, a mostly useless plugin, but one that’s useful to diagnose how plugins work.

Adding a Plugin to Your Project

Adding a plugin to your project is the same as adding any other Composer package to your project. Create a new folder

Since I didn’t create a packagist.org listing for this package, we need the repositoriessection. This tells Composer to use the git (vcs/version control system) repository at the provided URL as a repository.

This Composer file in place, there’s one last thing we’ll want to do before we run composer install. In a separate terminal window, run the following.

$ touch /tmp/composer.log
$ tail -f /tmp/composer.log

This creates our composer.log file, and then tails it. Tailing a file means showing the last few lines of output. When we run tail with the -f option, we’re telling tail to show us the last line of the file whenever the file is changed. This is a decades old technique for monitoring log files in the *nix world.

A Composer plugin developer can, (via the Composer\Script\Event object Composer passed to our handler or the Composer\Composer object Composer passes to the activemethod), examine and change Composer’s state at run time, implementing all sorts of extra functionality whenever a Composer project updates.

Covering that functionality in full is beyond the scope of this article, but with Composer being open source, there’s nothing stopping you from diving right in.

What Makes a Package a Plugin?

As we mentioned earlier, a Composer plugin is just a standard Composer package. However, it’s a standard Composer package with a specialcomposer.json file. Let’s take a look at our plugin’s composer.json file.

points to our plugin class (Pulsestorm\Composer\Example\Plugin). Since Composer will need to instantiate this class, that means you’ll need something in your autoload section that ensures PHP will load the class definition file. In our case, we used a standardPSR-4 autoloader

We see the composer-plugin type-tag our command line searching found, as well as the required extra configuration that configures theMagentoHackathon\Composer\Magento\Plugin class as a plugin.

Without getting into the specific technical details, this is the plugin that installs those extra files at the root level, above the vendor folder. In short, theMagentoHackathon\Composer\Magento\Plugin will

Listen for composer install and composer update events

Look at the extra->map section(s) for anycomposer.json file in the just installed or updated composer vendor packages

Use that information to copy file from the installed package, to the root level project folder

If that didn’t make sense, let’s walk through it. First, let’s find any composer.json files with a "map" section.

This is how the non-vendor files Magento needs to operate get from Magento core vendorpackages into the root folder of your project.

History of magento/magento-composer-installer

Before we wrap up, it’s worth noting that the magento/magento-composer-installerComposer plugin is a fork of the original Magento 1 Composer installer plugin built at a Magento hackathon, promoted by Firegento, and maintained by Daniel Fahlke. The original goals of this plugin were to build a system that allowed developers to use Composer to fetch Magento 1 plugins into vendor, and then install them into a Magento 1 system via a number of different strategies. This was necessary since Magento 1 never officially adopted Composer.

The Magento core team has repurposed the project as an automatic installer which, on one hand, shows the power and usefulness of open source. On the other hand, if you’re not familiar with the project history and you start exploring the plugin’s implementation in

However, if you keep the project’s original goals in mind, the source should make a little more sense.

Consequences

Between this article and last week’s, you should have a pretty good understanding of where all Magento’s files come from in a Composer “meta-package” installation. Understanding this is critical for Magento 2 developers and consultants looking to use and develop day-to-day on Magento’s systems. For example, it may be temping to drop some functions into

app/functions.php

as a shortcut way to get your code into a Magento system, but once you understand that any future composer update will wipe these changes away, the true cost of such short cuts become apparent. Don’t edit the core is as true as it ever was, but with Magento 2 it’s not always clear what is, and what isn’t, a core Magento file.

Next time we’ll be diving into Magento 2’s component system — the system that makes Composer distribution possible.

As Magento 2 approaches its first half-birthday, one thing is clear: Magento 2 is leaning heavily on PHP Composer for its developer workflow, and for the merchant facing Marketplace.

If you’re a developer working with Magento, you may be familiar withrepo.magento.com. This is Magento 2’s composer repository, and up until Magento Imagine 2016, its main purpose was to provide modern PHP developers with a way to install Magento 2 using PHP composer.

In this article, we’re going to show you how you can mirror your Magento specific composer packages on a local server. While not necessary, this is a useful precaution to take if you want to avoid any unscheduled maintenance bringing down your deployment and development pipelines. Along the way, we’ll also discuss Marketplace’s new composer based architecture, and end up touching on many lesser known composer features.

The package you want is at [THIS Git/SVN/Mercurial URL] and I found an archive at[THIS URL]

Packagist is a composer repository. One thing that catches a few composer newcomers by surprise is packagist, and other composer repositories, don’t actually host any packages. They just point to the location of a package on another server.

Other Repositories

Packagist support is baked into composer. However, it’s possible to point composer atother Composer repositories with a composer.json configuration that looks something like this

This flag ensures the magento/project-community-edition package is downloaded from repo.magento.com and not packagist.

The meta-package method downloads and installs Magento components (modules, themes, language packs, and libraries) into the vendor/magento/ folder. Thanks to Magento’sregistration.php file and PSR-4 support, there’s no longer a strictly defined place where Magento modules, themes, language packs and libraries need to be located. These components can now be separated out into individual composer packages.

The key to understanding Magento’s need for credentials is Magento Marketplace.

Magento Marketplace

Magento Marketplace is Magento 2’s replacement for the old Magento Connect. Magento Connect only hosted free Magento modules — commercial module listings pointed off towards independent software vendor’s websites where you could purchase the extension or service directly from the independent vendor.

Magento Marketplace changes that. Marketplace has free extensions available, but it also provides a one-stop shop for purchasing your Magento extensions. Once purchased, you can download the extension package from the My Account section of Magento’s website.

In addition to this download, the extension will also be available via the Magento 2 Component Manager in Magento’s backend, at System -> Web Setup Wizard. The Component Manager is a GUI for installing composer packages from repo.magento.com. It turns out that repo.magento.com is a session-ed packagist repository.

When you log in to repo.magento.com using the aforementioned HTTP Auth credentials, Magento’s composer repository returns a custom list of packages for you to install. This will include

The base Magento 2 CE packages

Any Magento 2 EE versions your account has access to

Any package you’ve purchased

This is what enables you to fetch purchased Magento Marketplace packages via Component Manager. You can also simply add new packages to your composer.json file and then update your system via the command line (i.e. composer update)

Mirroring repo.magento.com

While Magento’s adoption of composer is welcomed, it does add an additional wrinkle to deploying Magento 2 projects. The repo.magento.com repository is a new, single point of failure that, when down, could block a composer based deployment from being updated, or prevent your development team from getting started on a new project. Additionally, unlike the other (still not great) single points of failure in a composer project (packagist.org, github.com, etc.), repo.magento.com isn’t yet a battle tested system.

Because of this, it makes sense to create a local mirror of repo.magento.com. In addition to protecting you from unplanned maintenance/downtime at repo.magento.com, hosting your Magento packages on your local network (or even local development machine) should speed up Magento deployment tremendously.

Mirroring with Satis

The composer project has a second, sibling project called satis. Satis was created to allow developers to create their own local mirrors of packagist.org content. It turns out a stock composer repository requires zero dynamic processing — a repository is just a collection of static json files, and (optionally) mirrored archive packages.

To use satis, you’ll need to clone the GitHub repository to your local development machine with one of the following commands

Satis will output packages.json files and (optionally) package archives to the output folder you specify. You’ll be able to upload these files to any simple web server, and have a working packagist repository.

The JSON configuration file is where you tell satis which repositories you’d like to mirror, which packages in the repositories, as well as any other satis configuration needed.

The syntax for building a satis mirror is

$ php bin/satis build config.json build-folder

The file we’ve named config.json above is often named satis.json by convention.

Satis Configuration

Next up, we’re going to create a satis configuration file and review its options. While the composer manual has a section on satis, the file format isn’t as well documented as it could be. The following is one possible configuration, please get in touch or comment below if you see something egregiously wrong here — I’m learning with the rest of you here!

The name property should be a simple description of your repository. It’s used in a static HTML file generated for the repository, so don’t use a name that would embarrass your mother or your supervisor.

The homepage property is the URL you’re planning on hosting your satis repository at. This can be a URL on the Internet, or a URL local to your network/dev machine. This URL will be used in a static HTML file generated for the repository, so make sure its accurate.

The repositories property is the first vital configuration field. This should be an array of the composer repositories you’d like to mirror with satis. In our case, this is the composerrepository at repo.magento.com. Other types of repositories you might see are VCS orpear.

The require-dependencies and require-dev-dependencies flags make sure composer will require any dependencies a specific package may have. Not strictly necessary in our case (see require-all below) but it never hurts to be explicit.

The require-all property tells satis we want to grab every package on the repository. Since our goal is to create a complete local mirror, we want this here. If we were interested in creating a mirror of select packages, we’d use a require property and list out the packages.

The final top level archive property tells satis that, in addition to creating apackages.json file for us that points to packages, we’d like satis to grab and/or build a project archive for us as well.

In the archive object configuration, the directory property tells satis where the archived files should be copied to (i.e. build-folder/dist), the format property tells satis which archive format we should use, the prefix-url should be, again, the URL for your repository, and setting the skip-dev option to false ensures we get every file in a package.

The prefix-url key is important here, as it’s the URL satis will use in the generatedpackages.json file to point to our local mirrored archive.

Building the Mirror

Once you’ve created your satis.json, you can create the mirror with the following

$ php bin/satis build satis.json public -vvv

If your repo.magento.com HTTP credentials aren’t stored in ~/.composer/auth.json, satis will prompt you for them. The -vvv flag is optional, but will ensure satis is verbose in its output. This can help point to problems in your configuration, or with your network connection.

When the command finishes running, you’ll want to upload the files in public to the web root of whatever URL you configured in satis.json (composer.pulsestorm.dev in our case).

When satis is done running, you can take a look at the repository files using the unix findcommand

You should see archives of all the community edition files, any EE files your account license grants you access to, and any extension files you’ve purchased. Upload the entire contents of the public folder to your web server, and you’ll have a local composer repository up and running.

With our mirror created, we’re ready to install Magento 2.

Installing Magento 2 with our Mirror

Normally, when installing Magento via composer, you use the following command

You’ll see that its composer.json file points to repo.magento.com. The --repository-url option only applies to the package that create-project grabs. Otherwise, composer will use whatever it finds in the project’s base composer.json file.

This is easy enough to work around — all we need to do is use the no-install flag — this way create-project will only download and extract the project, it won’t run composer install. This will give us a chance to edit the composer.json file before running install.

Installing Magento 2

OK! We’re ready. Step 1, lets clear our composer cache

$ composer clear-cache

This is not strictly necessary, but useful when you’re first setting up a mirror, and may have an invalid package reference stashed in cache somewhere.

Also, if your mirror’s not located on an https server, you may need to set composer’s global secure-http flag to false.

$ composer global config secure-http false

Recent versions of composer will refuse to run over non-encrypted HTTP.

Again, the -vvvs are optional, but viewing composer’s verbose output can help us ensure that no package was/is downloaded from repo.magento.com. When composer’s done running, change into the project-community-edition folder, and take a look atcomposer.json

and composer will grab all Magento’s packages from your local mirror — no access torepo.magento.com required.

Wrap Up

All of this, of course, is only a start to keeping and maintaining a local composer mirror. EE users will want to check their enterprise license agreement to make sure doing this falls within acceptable use of the EE source code, and regardless of which version of Magento you’re running you’ll want to make sure your mirror isn’t located anywhere online, as you may become an inadvertent distribution point for commercial extensions you don’t have the right to distribute.

You’ll also want to figure out a way to get satis running on a regular basis — otherwise you may miss important updates to the Magento repository. The very bold may want to expand their mirroring to packagist itself, but that’s a larger problem filled with all sorts of blind alleys and large hard drives.