mc_project_2 / project settings regitry APIV2

This is a Corpus Paas reactor, deploy your projects with style, salt style.

This can either:

  • Deploy locally a project
  • Deploy remotely a project over ssh (the remote host must have makina-states installed)
mc_states.modules.mc_project_2.clean_salt_git_commit(directory, commit=True, **kw)
mc_states.modules.mc_project_2.deploy(name, *args, **kwargs)

Deploy a project

Only run install step:

salt-call --local -ldebug mc_project.deploy <name> only=install

Run only one or certain install step:

salt-call --local -ldebug mc_project.deploy\
        <name> only=install only_steps=00_foo
salt-call --local -ldebug mc_project.deploy\
        <name> only=install only_steps=00_foo,02_bar

Only run install & fixperms step:

salt-call --local -ldebug mc_project.deploy <n> only=install,fixperms

Deploy entirely (this is what is run whithin the git hook):

salt-call --local -ldebug mc_project.deploy <name>

Skip a particular step:

salt-call --local -ldebug mc_project.deploy <name>\
        skip_release_sync=True skip_archive=True skip_notify=True
mc_states.modules.mc_project_2.get_configuration(name, *args, **kwargs)

Return a configuration data structure needed data for the project API macros and configurations functions project API 2

name
name of the project
remote_less
Does the project use local remotes (via git hooks) for users to push code inside remotes and have the local working copy synchronized with those remotes before deploy. Default to True, set to False to use local remotes If the project directory .git folder exists, and there is no local remote created, the local remotes feature will also be disabled
fqdn
fqdn of the box
minion_id
minion_id of the box
default_env
environnemt to run into (may be dev|prod, better to set a grain see bellow)
project_root
where to install the project,
git_root
root dir for git repositories
user
system project user
groups
system project user groups, first group is main
defaults
arbitrary data mapping for this project to use in states. It will be accessible throught the get_configuration().data var
env_defaults
per environment (eg: prod|dev) specific defaults data to override or merge inside the defaults one
os_defaults
per os (eg: Ubuntu/Debian) specific defaults data to override or merge inside the defaults one
only_install
Only run the install step (make the others skipped)
skip_archive
Skip the archive step
skip_release_sync
Skip the release_sync step
skip_install
Skip the install phase
skip_rollback
Skip the rollback step if any
skip_notify
Skip the notify step if any

Any other kwarg is added to the data dict.

Internal variables reference

pillar_root
pillar local dir
salt_root
salt local dir
archives_root
archives directory
data_root
persistent data root
project_git_root
project local git dir
pillar_git_root
pillar local git dir
nodata
do not compute data
data
The final mapping where all defaults will be mangled. If you want to add extra parameters in the configuration, you d better have to add them to defaults.
force_reload
if the project configuration is already present in the context, reload it anyway
sls_includes
includes to add to the project top includes statement
no_default_includes
Do not add salt_minon & other bases sls like ssh to default includes
rollback
FLAG: do we rollback at the end of all processes

You can override the non read only default variables by pillar/grain like:

salt grain.setval makina-projects.foo.url 'http://goo/goo.git
salt grain.setval makina-projects.foo.default_env prod

You can override the non read only default arbitrary attached defaults by pillar/grain like:

/srv/projects/foo/pillar/init.sls:

makina-projects.foo.data.conf_port = 1234
mc_states.modules.mc_project_2.get_configuration_item(project, item=<object object>, **kw)

Return an item, maybe filtered or all the config

in the form:

{key: <itemKey or 'cfg'>, item: VALUE, cfg: <Whole config dict>}

CLI examples:

salt-call --local                mc_project_2.get_configuration_item eee git_deploy_hook
salt-call --local                mc_project_2.get_configuration_item eee
mc_states.modules.mc_project_2.get_project(name, *args, **kwargs)

Alias of get_configuration for convenience

mc_states.modules.mc_project_2.init_pillar_dir(directory, init_data=None, user=None, group=None, project=None, commit_all=False, do_push=False, remote_less=False, **kw)

Initialize a basic versionned pillar directory

directory
directory to execute into
user
user to exec commands as and for new files
group
group for new files
commit_all
do we do a final git add/commit
do_push
do we do a final push
project
project name
init_data
configuration options in the corpus format (see get_configuration)
mc_states.modules.mc_project_2.init_project(name, *args, **kwargs)

See common args to feed the neccessary variables to set a project You will need at least:

  • A name
  • A type
  • The pillar git repository url
  • The project & salt git repository url
mc_states.modules.mc_project_2.init_remote_structure(host, project, **kw)

Initialize a remote project structure over ssh

CLI Examples:

salt-call --local mc_project.init_remote_structure host.fr <project>
mc_states.modules.mc_project_2.init_repo(working_copy, user=None, group=None, ret=None, bare=True, init_salt=False, init_pillar=False, init_data=None, project=None, remote_host=None, cfg=None, remote_less=False, api_version='2')

Initialize an empty git repository, either bare or a working copy

This can be either:

  • a basic corpus-salt base project template
  • a pillar repo (containing an init.sls)
  • an enmpty directory (containing an empty .empty file just for git repo init)

CLI Examples:

salt-call --local mc_project.init_repo /foo
salt-call --local mc_project.init_repo /foo bare=true
salt-call --local mc_project.init_repo /foo init_salt=True
salt-call --local mc_project.init_repo /foo init_pillar=True
salt-call --local mc_project.init_repo /foo bare=true init_salt=True
salt-call --local mc_project.init_repo /foo bare=true init_pillar=True
mc_states.modules.mc_project_2.init_salt_dir(directory, user=None, group=None, commit_all=False, do_push=False, project=None, init_data=None, remote_less=False, **kw)

Initialize a basic corpus project directory

directory
directory to execute into
user
user to exec commands as and for new files
group
group for new files
commit_all
do we do a final git add/commit
do_push
do we do a final push
project
project name
init_data
configuration options in the corpus format (see get_configuration)
mc_states.modules.mc_project_2.init_ssh_user_keys(user, failhard=False, ret=None)

Copy root keys from root to a user to allow user to share the same key than root to clone distant repos. This is useful in vms (local PaaS vm)

mc_states.modules.mc_project_2.install(name, only_steps=None, task_mode=False, *args, **kwargs)

You can filter steps to run with only_steps All sls in .salt which are not special or tasks (beginning with tasks will be executed)

eg:

salt mc_project.deploy only=install onlystep=00_foo,001_bar
salt mc_project.deploy only=install onlystep=00_foo

Add the link wired in salt folders (pillar & salt) & register the pillar in pillar top

name
list of project(s) separated by commas

Link a salt managed directory into salt root

This takes care of not leaving a dangling symlink

Add the link wired in pillar folder & register the pillar in pillar top

name
list of project(s) separated by commas

Link a salt managed directory into salt root This takes care of not leaving a dangling symlink

mc_states.modules.mc_project_2.load_sample(cfg, *args, **kwargs)

Load the project PILLAR.sample back to the cfg configuration dict

mc_states.modules.mc_project_2.orchestrate(host, project, init=True, init_project=None, init_pillar=None, init_remote=True, sync=True, sync_project=None, sync_pillar=None, refresh=True, refresh_pillar=None, refresh_project=None, deploy=True, pillar_origin=None, pillar_rev=None, origin=None, rev=None, pre_init_hook=None, post_init_hook=None, pre_init_remote_hook=None, post_init_remote_hook=None, pre_sync_hook=None, deploy_fun='mc_project.remote_deploy', deploy_kwarg=None, deploy_args=None, post_sync_hook=None, pre_hook=None, post_hook=None, only=None, only_steps=None, **kw)

Orchestrate a project deployment, remotely

Note:

a project is composed by it’s code & deployment recipe (.salt) and it’s pillar.

This:

  • Run the pre init hook if any
  • initiliazes & prepare the code locally
  • Run the post init hook if any
  • Run the pre init remote hook if any
  • May initiliazes the remote project structure
  • Run the post init remote hook if any
  • Run the pre sync hook if any
  • Sync the code to remote buffer directories
  • Sync the remote deployment directories (pillar & project)
  • Run the post sync hook if any
  • Run the pre deploy hook if any
  • Run the deployment procedure (salt-call mc_project.deploy dance)
  • Run the post deploy hook if any
The deployed code will at first be initialized:
project code is initialized as either:
  • from an empty structure (shell helloworld)

  • from git:

    origin

    git url

    rev

    git remote tag/branch/changeset

  • from an empty structure (shell helloworld)

pillar code is initialized as either:
  • from an empty structure (shell helloworld)

  • from git:

    pillar_origin

    git url

    pillar_rev

    git remote tag/branch/changeset

If the urls were not specified, but the git repositories present in the local directories are valid, they will be deployed as-is.

If the git repositories have a valid remote, the sync step will use it, even if the “origin/pillar_origin” were not specified explicitly

host
host where to deploy
project
project to deploy onto (name)
init/init_project/init_pillar
do we do the full init step do we do the init_project step (overrides init). do we do the init_pillar step (overrides init).
init_remote
do we do the init_remote step
origin/pillar_origin
url of the pillar to deploy if any (if None: empty pillar)
rev/pillar_rev
changeset if the project/pillar is from a git url (master)
sync/sync_project/sync_pillar
do we do the full sync step. do we do the sync_project step (overrides sync). do we do the sync_pillar step (overrides sync)
refresh/refresh_project/refresh_pillar
do we update the code prior to sync. do we do the refresh_project step (overrides refresh_project). do we do the refresh_project step (overrides refresh)
deploy
do we do the deploy step
deploy_args
any extra deploy/run_task positional argument
deploy_kwarg
any extra deploy/run_task extra argument
pre_init_hook/post_init_hook/ pre_init_remote_hook/post_init_remote_hook/ pre_sync_hook/post_sync_hook/ pre_hook/post_hook
deployment hook (see above lifecycle explaination & hook spec)
only/only_steps
mc_project.deploy deploy limits arguments(if any)

An hook is a salt function, in any module with the following signature:

def hook(host, project, opts, **kw)::
    print("hourray")

Making in a execution module, /srv/salt/_module/foo.py:

def hook(host, project, opts, **kw)::
    print("Called on {0}".format(host))

Can be called like this:

salt-call --local mc_project.orchestrate h proj init_hook=foo.hook

CLI Examples:

salt-call --local mc_project.orchestrate host.fr <project>\
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project>\
        origin="https://github.com/makinacorpus/corpus-pgsql.git"\
        rev=stable\
        pillar_origin="https://github.com/mak/corpus-pillar.git"\
        rev=stable
salt-call --local mc_project.orchestrate host.fr <project>\
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project>\
        deploy_args=['a'] \
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project>\
        deploy_kwarg={'myparam': 'a'} \
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
mc_states.modules.mc_project_2.orchestrate_task(host, project, task, init=True, init_project=None, init_pillar=None, init_remote=True, sync=True, sync_project=None, sync_pillar=None, refresh=True, refresh_pillar=None, refresh_project=None, deploy=True, deploy_args=None, deploy_kwarg=None, pillar_origin=None, pillar_rev=None, origin=None, rev=None, pre_init_hook=None, post_init_hook=None, pre_init_remote_hook=None, post_init_remote_hook=None, pre_sync_hook=None, post_sync_hook=None, pre_hook=None, post_hook=None, only=None, only_steps=None, **kw)

Orchestrate a project through mc_project._orchestrate and use mc_project.remote_task <TASK NAME>

CLI Examples:

salt-call --local mc_project.orchestrate_task host.fr <project> task_make_users \
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project> make_users\
        origin="https://github.com/makinacorpus/corpus-pgsql.git" task_make_users\
        rev=stable\
        pillar_origin="https://github.com/mak/corpus-pillar.git" task_make_users\
        rev=stable
salt-call --local mc_project.orchestrate host.fr <project> task_make_users\
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project> task_make_users\
        deploy_args=['a'] \
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
salt-call --local mc_project.orchestrate host.fr <project> task_make_users\
        deploy_kwarg={'myparam': 'a'} \
        origin="https://github.com/makinacorpus/corpus-pgsql.git"
mc_states.modules.mc_project_2.push_changesets_in(directory, remote='origin', branch='master:master', opts='', **kw)

Thin wrapper to git push

directory
directory where to act on
user
user to push as
branch
branch part of the git command
remote
remote to push to
opts
‘origin’, opts to give

CLI Examples:

salt-call --local  mc_project.push_changesets_in /foo
salt-call --local  mc_project.push_changesets_in /foo opts="-f"
salt-call --local  mc_project.push_changesets_in /foo opts="-f"
mc_states.modules.mc_project_2.remote_deploy(host, project, *args, **kw)

Run a deployment task, remotely

host
host to deploy onto
project
project to deploy
deploy_kwarg
deploy kwargs
deploy_args
deploy arguments
deploy_only_steps/only_steps
set only_steps kwarg (shortcut)
deploy_only/only
set only kwarg (shortcut)
salt_function
salt deploy function (one of: mc_project.deploy/mc_project.run_task)

CLI Examples:

salt-call --local mc_project.remote_deploy                 host.fr <project>
salt-call --local mc_project.remote_deploy                 host.fr <project> only=install,fixperms
salt-call --local mc_project.remote_deploy                 host.fr <project> only=install,fixperms only_steps=0.sls
mc_states.modules.mc_project_2.remote_project_hook(hook, host, project, opts, **kw)

Execute an hook

An hook is a salt function, in any module with the following signature:

def hook(host, project, opts, **kw)::
    print("hourray")
mc_states.modules.mc_project_2.remote_run_task(host, project, task=None, *args, **kw)

Run a task from the .salt directory, remotely

Task can be either given as a kwarg or the first positional argument.

CLI Examples:

salt-call --local mc_project.remote_run_task                 host.fr <project> task=task_helloworld.sls
salt-call --local mc_project.remote_run_task                 host.fr <project> task_helloworld.sls
salt-call --local mc_project.remote_run_task                 host.fr <project> task_helloworld.sls a=b
mc_states.modules.mc_project_2.remote_task(host, project, *args, **kw)

Alias to remote_run_task

CLI Examples:

salt-call --local mc_project.remote_task                 host.fr <project> task=task_helloworld.sls
salt-call --local mc_project.remote_task                 host.fr <project> task_helloworld.sls
salt-call --local mc_project.remote_task                 host.fr <project> task_helloworld.sls a=b
mc_states.modules.mc_project_2.remove_path(path)

Remove a path.

mc_states.modules.mc_project_2.report()

Get connection details & projects report

CLI Examples:

salt-call --local mc_project.report
mc_states.modules.mc_project_2.rollback(name, *args, **kwargs)

Run the rollback corpus step

mc_states.modules.mc_project_2.rotate_archives(name, *args, **kwargs)

Run the rotate_archives corpus step

mc_states.modules.mc_project_2.run_task(name, only_steps, *args, **kwargs)

Run one or more tasks inside a project context.

You can filter steps to run with only_steps

All sls in .salt which are a task (all files beginning with task_ will be searched and the one matching only_steps (string or list) will be executed)

mc_states.modules.mc_project_2.set_configuration(name, cfg=None, *args, **kwargs)

set or update a local (grains) project configuration

mc_states.modules.mc_project_2.sync_git_directory(directory, origin=None, rev=None, sync_remote='sync', refresh=False, local_branch=None, **kw)

Agressively sync a git working copy with its remote

directory
directory where to act
origin
remote origin <url>
rev
changeset to deploy
local_branch
local branch to set to
sync_remote
name of the remote
refresh
if the working copy exists, force the git pull dance

CLI Examples:

salt-call --local \
        mc_project.sync_git_directory /foo origin=git://foo rev=develop
mc_states.modules.mc_project_2.sync_hooks_for_all(*args, **kwargs)

Get connection details & projects report

mc_states.modules.mc_project_2.sync_modules(name, *args, **kwargs)

Install custom modules to the salt modules directory in the global module dir,

Note: we install two symlinks to provide room for overlapping modules eg: project1/project/.salt/_modules/foo.py:bar can be called:

salt-call foo.bar()
salt-call foo_foo.bar()
mc_states.modules.mc_project_2.sync_remote_working_copy(host, directory, remote_directory=None, remote_local_copy=None, lremote=None, **kw)

CLI examples:

salt-call --local mc_project.sync_remote_working_copy\
        a.fr /srv/remote-projects/a.fr
salt-call --local mc_project.sync_remote_working_copy\
        a.fr /srv/remote-projects/a.fr /otherdir
mc_states.modules.mc_project_2.sync_working_copy(wc, rev=None, ret=None, origin=None, reset=False, **kw)

Synchronnze a directory with it’s git remote

directory
directory to execute into
reset
force sync with remote
user
user to exec commands as and for new files
rev
force rev to reset to
origin
origin to sync with
mc_states.modules.mc_project_2.uncache_project(name)

Uncache a configuration after an initial load either by it’s name or by its structure

Remove the link wired in salt folders (pillar & salt)

name
list of project(s) separated by commas

Remove the link wired in pillar folder & unregister the pillar in pillar top

name
list of project(s) separated by commas

Remove the link wired in salt folder & unregister the pillar in pillar top

name
list of project(s) separated by commas
mc_states.modules.mc_project_2.working_copy_in_initial_state(wc, **kw)

Test if a directory is at the first git commit from this system

wc
where to execute
user
user to act with