Sunday, September 18, 2011

Using rsync to copy/sync two file systems

I've used the rsync command a number of times to synchronize a couple files systems. It's particularly useful for doing backups of one set of files to another. These are some EXTREMELY loose notes about how I did things a couple times that I found in old emails.

Example 1: Replicate on directory to another place on same machine.
cd /data
rsync -a dir1 /dest

Copies all the files from /data/dir1 and drops them in /dest/dir1.

Example 2: Doing on windows with cygwin between two separate windows boxes. Very loose instructions, but a long time ago I was able to make this work between two windows boxes by using a client/server model (e.g. remotely connecting between the two boxes.)

  • install cygwin
  • Map \\axis\backup as X:
  • mkdir -p /cygdirve/x/LaptopBackup/Users-rmills
  • cd C:/Users
  • rsync -a rmills /cygdrive/x/LaptopBackup/Users-rmills
Effectively copies everything from my /Users/rmills directory to the backup directory.

Example 3: Starting from scratch and copying a lot of stuff

On a windows server box, setup K: in /rmills/rsync
  • Create rsync.conf
  • Create rsync.scrt
  • On Server:
    • rsync --verbose --daemon --config=./rsync.conf --no-detach
    • (Take away --no-detach after prove it works.)
  • On Client:
    • rsync -progress -av SomeDirectory rmills@server::vol1
    • (consider "--update" to see if it just copies a few files)
 This basically allows two rsync processes to reach out to each other and copy files.

Friday, September 16, 2011

Parameterizing deployment of ASP applications with VS and MSDeploy

I've worked with a few ASP applications recently where I've wanted to be able to custom-configure the Web Config upon deployment depending on whether it's going to a developer's machine, test server, or production environment. In particular, I'm working on an application now that needs to pull data via SOAP or WCF from a neighboring SharePoint server. The URL of this server will, of course, be different depending on the environment.

There are a variety of situations I wanted to handle.
  1. Local deployment and testing on a development rig
  2. Fully scripted deployment using Web Deploy through a continuous integration server onto our development server
  3. On-demand deployment using the IIS "Deploy > Import Application" wizard
I more-or-less ignored case #1 because this is what developers do every day and they just hack whatever the way to make it work. (Sigh.) I did, however, want to make sure that the parameterization I did continued to work in their environment(s).

For case #2, I've been using msdeploy.exe via the App.deploy.cmd script that appears when you do "Build Deployment Package" from Visual Studio. This can be invoked from a script by simply doing MSBuild /t:Package during build. I'll leave the dynamics of that out of scope for this post. I'm more interested in how you set up the "SetParameters.xml" file properly so that you can feed MSDeploy some scripted parameters during installation to get the Web.Config the way you want it.

For case #3, I've been using the Deploy > Import Applications wizard to pull in the ZIP file produced by the "Package" step of #2. This works well and has a "Enter Application Package Information" page/panel for setting application parameters such as "Application Path" and "Connection String." I wanted to find a way to add parameters to this panel requesting other values.

As it turns out, there are a two primary techniques for building these kinds of configurations. The first is to use "Configurations" in Visual Studio to create additional Web.Config transforms that will substitute values that can be compiled into the applications for different environments. This works well if you know (ahead of time) what the parameters will be in all your configuration environments.

Build Configuration for Web.config

To make this work, you can first create a visual studio "Configuration" in Build > Configuration Manager. Then, create a "New..." configuration.




 I created one called "Staging"


Then, in your Solution Explorer, you use "Add Config Transformations" on the Web.config file.



 This will look at your configurations and create a Web.config transformation file for any new Configurations you have.



 Open your new Web.staging.config file and add an entry:

 
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <appSettings>
     <add key="MySetting" value="My Value"
         xdt:Transform="Replace" xdt:Locator="Match(key)"/>
  </appSettings>
</configuration>

Now, when you select the "Staging" configuration and do a "Build Deployment Package" from Visual Studio, it will automatically replace the setting in your master Web.config with this value. You may have to use Build > Configuration Manager to select the build configuration you want to use (e.g., Staging). You can deploy this package an it will automatically be set properly. This is especially useful for Database connection strings that tend to be fairly well known in most development and testing environments, as long as you're not worried about checking those values into your version control system. I used the following for a DB connection string for the ASP application we were building:

 
  <connectionStrings>
    <add name="ApplicationServices"
      connectionString="data source=TestDBServer;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|\aspnetdb.mdf;User Instance=true"
      xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
  </connectionStrings>

Parameters.xml

The second way to parameterize your web config is to establish a "parameters.xml" file for your application that creates these transforms in an externalized way that can be set at deployment time. This is the technique that actually works for cases #2 and #3 above. By setting up parameters.xml, I get a entry to appear in the SetParameters.xml file as well as in the deployment wizard within IIS manager. I'll focus on that for the rest of this post.

First, create a new XML file called "parameters.xml" and store it in the root of your project. Put the following entries to modify your parameters:


 
 <parameters>
  <parameter name="External SP Server Url"
             description="URL of external SharePoint server. Example: http://externalsp.portal.local "
             defaultValue="http://localhost"
             tags="">
    <parameterEntry
        kind="XmlFile"
        scope="obj\\Debug\\Package\\PackageTmp\\Web\.config$"
        match="//appSettings/add[@key='Portal.ExternalSPServerUrl']/@value" />
  </parameter>
</parameters>

This will generate the proper "External SP Server Url" entry in your SetParameters.xml file for scripting as well as place it on the wizard when you deploy with IIS. This replaces a key="Portal.ExternalSPServerUrl" in configuration/appSettings section of the web.config. It took me a while to get the "match" criteria set properly to find it and set the value.

Both these solutions work together -- parameters.xml and Web.config configuration files. One thing to note is that the "defaultValue" in the parameters.xml file will trump whatever settings you have in the web.config templates, so you need to provide a proper value during installation.

References

I found the following links valuable in figuring this out:
  • http://msdn.microsoft.com/en-us/library/ff398068.aspx - how to use parameters to configure deployment
  • http://vishaljoshi.blogspot.com/2009/03/web-deployment-webconfig-transformation_23.html - web config transformation using configurations
  • http://msdn.microsoft.com/en-us/gg454290 - transforming web config for deployment

Thursday, September 15, 2011

Deploying ASP web application with command line msdeploy without deleting all files on server

.
This mimicks the IIS wizard prompt "No, just append the files in the application package to the destination" on the "Overwrite Existing Files" panel. I used the "-enableRule:DoNotDeleteRule" for msdeploy.exe to accomplish this.


First, Use "Package" option within Visual Studio 2010 or via MSBuild.exe on the command line. This produces a "MyApp.deploy.cmd" along with a matching "MyApp.SetParameters.xml" that goes with your MyApp.ZIP file.

Then, simply invoke the CMD file to deploy the application using the "enableRule" parameter.

C:\>call MyApp.deploy.cmd /Y /m:myhost /u:my_user /p:my_password -enableRule:DoNotDeleteRule -debug -setParamFile:C:\MyPath\MyApp.SetParameters.xml

This will prevent configuration files such as web.config from being deleted as it deploys the application over top of the previous installation.

This has a few assumptions:
  1. You've already gone through the whole set of configuration plumbing needed to set up the MSDeploy command to properly work for remotely deploy the application to IIS server. 
  2. You already have things working such that you can deploy the ZIP file using IIS manager with Deploy > Import Application and stepping through the wizard.

Friday, June 3, 2011

Sync multiple google calendars on iPhone

I find it annoying that every time my wife adds another calendar to her google account, I have to spend 10 minutes googling (!) to find some secret link that you have to surf to to select which calendars appear for sync. I'm not sure why this link is not on the first hit any time you search for this.

https://www.google.com/calendar/iphoneselect

Anyway, surf here with safari on you iPod or iPad or iPhone, pick the calendars you want to appear, then magically they will be selectable on your apple device.

Recorded for myself and posterity.

Tuesday, May 24, 2011

Finding active file in Vistual Studio 2010 solution explorer

I often use Ctrl-, (control comma) or another search variant to locate files when I'm working in Visual Studio 2010. Unfortunately, it often leaves me with a bunch of open windows and no easy way to navigate to the project, directory, or other tree area within my solution where the file actually lives. The pop up search box often gives the path, but it is still a nuisance to go expand a bunch of "+" signs to go find it. I knew there had to be an easy way.
As it turns out, there is. Visual Studio has an option (Tools > Options > Projects and Solutions > General) called "Track Active Item in Solution Explorer" that will automatically update the solution explore to hop to the file that is active.
The downside, of course, is that it ALWAYS does this. I originally found the feature via another post that tells how to create a macro to do this on-demand: [locate-file-in-solution-explorer-visual]. Now I just have to figure out how to add a macro to vStudio. :-)

Happy coding.

Wednesday, May 4, 2011

Working with XML and XSD schemas in Visual Studio

I recently started working with XML files as part of a SharePoint project I was working on in Visual Studio. As the project evolved, I started creating some crazy XML files that defined the structure of a SharePoint site. It wasn't long before the crazy schema I invented on the fly started become confusing. Worse, I started forgetting the structure of the elements I had invented and had to start looking through the source code that processed it to try to remember what the structure and options were supposed to be.

Better solution: Create an XSD file with the structure definition in it. Basically, I followed these steps.
  1. Reverse engineer the XSD from my XML file
  2. Annotate the XSD file with some documentation about what it all meant
  3. Add a reference to the XSD in the XML file
  4. Continue editing the XML file in Visual Studio
Reverse Engineering the XSD 
First step was to create an XSD. I didn't want to type it all by hand, so I found this link: http://www.dotkam.com/2008/05/28/generate-xsd-from-xml/. It referenced a tool called "Trang" which processes and XML file and produces an XSD. Download it here.

I created a simple batch file:
@echo off
: Batch file to invoke Trang to convert an XML file to a XSD
set JAVA_HOME=C:\apps\jdk1.6.0_20
set TRANG_HOME=C:\apps\trang
%JAVA_HOME%\bin\java.exe -jar %TRANG_HOME%\trang.jar %*
 Then I converted my existing XML file:
xml2xsd.bat myFile.xml schema.xsd

And it produced a reasonable XSD file that I could begin using. What's nice, is if you load this file into Visual Studio, it provides some nice tools for exploring and viewing your schema. (I have NOT found any good ways for actually editing the schema other than hacking on the XSD code.)

Annotate XSD Schema

Once you get the basic XSD file, it's nice to add some explanation to it. You can easily use the "xs:annotate" and "xs:documentation" elements to do that.

<xs:annotation>
<xs:documentation xml:lang="en">
Documentation about your element.
Hint: First line is shown as "documentation"
<xs:documentation>
<xs:annotation>


This allows you to self-document the XSD file so you can look at it later and remember what the hell you were thinking when you first made it. It also has the added benefit of showing up when you turn on "Show Documentation" in Visual Studio Schema Model view.

A quick note on Visual Studio: When you open an XSD file in Visual Studio, it shows you the "XML Schema Explorer." From there, you can select elements and "Show in Content Model" which allows you to expand elements and browse through your schema. There are other modes, too, but I haven't figured them out.

Add Reference to XSD in XML File

Once you have an XSD, it's nice to actually make use of it. You can do this by annotating the root element in your XML file:

<RootNode  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="MySchema.xsd">

This allows anything processing the XML file to actually find the XSD and determine whether your XML file is "correct."  In my case, my XSD file was in the same directory as the XML file, so I didn't have to post the XSD anywhere.

Continue Editing XML File in Visual Studio

After doing all this, you're back to editing your XML file. The nice side effect of all this in Visual Studio is that it activates Intellisense for XML editing. This means:
  1. It will suggest name completion as you start typing elements and attributes
  2. It will suggest elements and attributes that can be inserted in a location
  3. It will suggest errors when you type elements that don't exist
I immediately found that this helped me work with my XML files much more quickly without creating bad formatting errors.

Wednesday, February 2, 2011

Upgrade and clone hard drive in laptop

I recently bought an upgraded hard drive to put into my laptop.

Side note: Momentus XT 500 GB "hybrid" hard drive. This is a 7200 RPM drive with 4GB of solid-state disk internal to act as a write-through cache. It's supposed to drastically speed up the performance of the drive for executing common things (like booting, swapping, running apps).

Goal: clone current hard drive, swap, then boot.

Approach 1: purchase Apricorn EZ Gig product that comes with a cable and a software package to clone the drive. Price: $39. Unfortunately, they were out of stock locally, so I attempted the poor-man's approach.

Approach 2: Use an adapter kit I had laying around to hook up the drive to USB, then use CloneZilla to do it.


  1. Visit http://clonezilla.org/clonezilla-live.php and download clonezilla ISO. (I got the AMD64 arch for my E5500 laptop)
  2. Visit http://tuxboot.org/download.php so I can burn the ISO to a USB stick.
    1. Follow to source forge, go into the latest date directory, and download the tuxboot-windows.exe version.
  3. Run tuxboot.exe (it wants admin access to write to the device)
  4. Burn the image to the USB device. Don't worry about rebooting since you won't be booting to this machine immediately.
  5. Reboot onto the USB stick
  6. Follow prompts to start up clonezilla
  7. Choose drive-to-drive clone
  8. Follow prompts, pick the right drives for source and destination
  9. Let it churn. My machine was doing approximately 2-3 GB/min, so it took a few hours.
When it finished, I simply swapped the drives and the system magically booted up. Easy, peazy, lemon squeezy.