C# (CSharp) DataMigration Namespace

Classes

Name	Description
AbstractBaseExistence	A base class for existence objects that can use a C# HashSet for their implementation. Child classes usually just implement Init() to populate the HashSet, or they can leave it empty; plan-specific existence objects can use this mechanism to implement a first-writer-wins de-duping strategry on feed data items.
AbstractBaseLookup	A base class for lookups that can use a C# Dictionary object for their implementation. Child classes usually just implement Init() to populate the Dictionary.
BadRowException	BadRowExceptions are thrown by row processors when they absolutely can do nothing with a raw row of data. The data is included, and responsibility for handling it returns up the stack to the FeedProcessor.
BadRowsForeignIdDescriptor
BadRowsOutboundFeed	This copies the BadRows for a given integration loadnumber up to S3.
CompiledXforms	Compiled xforms. This is an instance of IMethodResolver, which has compiled C# functions that actually live in the DataMigration assembly. IMethodResolvers can be external, and they don't have to be compiled (although support for scripted methods hasn't been added yet).
CsvParser	Csv parser. Wraps the LumenWorks.Framework.IO Csv reader, published at http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader.
CsvTabDelimitedParser	For some feeds, the LumenWorks.Framework.IO Csv does a better job than the home-grown ParameterizedDelimiterParser. But...it turns out there's a bug in the Microsoft Xml deserialization code when trying to specify \t in the plan. So instead we just derive from the CsvParser here, and hardcode the \t.
DataMigration	The principal class for the DataMigration tool. It contains the Main() function, and processes command line arguments. It deserializes the DataMigrationPlan object, instantiates a MigrationEngine, and calls its Transform() method.
DataSourcePlan
DescriptorProperty
ExistenceDescriptor
FeedManager	FeedManager implements IFeedManager, an entity which, given an IFeedAccess instance can determine if feed work needs to be done, and move the files from CurrentToArchive, from Drop to Local, and download to the local file system. FeedManager works in tandem with IFeedAccess to manage this process.
FeedProcessor	There's an instance of FeedProcessor for each DataSource in the Plan. It uses the DataSourcePlan and the contained FeedFilePlans to figure out what files to process, which parsers to use, which Stager to use, etc.
FeedStagerDescriptor
FileSystemPublishMonitoringData
LookupDescriptor
MethodResolverDescriptor
NonFileBasedFeedManager	This is used for non-file-based data sources, it does no actual work, but just allows the MigrationEngine to go directly to using the Parser to do work.
ParameterizedDelimiterParser	ParameterizedDelimiterParser takes some sort of character property from the DataSourcePlan, and uses that to parse the file. Thus a crude CSV parser (putting aside complications inherent in csv) would specify a comma, where a tab-delimited-text parser specifies a tab.
ParserDescriptor
PhaseLogEntry	PhaseLogEntry is a class that encapsulates all the data we're writing in Phase Logging PhaseLogging is an activity that is done by the DataMigration tool to cope with the fact that phases of the process may be done by C#, but other phases (e.g. Sql Server - based PostProcessing modules) that will not be able to write to log4net. This will be consumed by IPhaseLogWriter implementations as a unified way to show what happened on a given run of the DataMigration tool.
PhaseLoggerDescriptor
PlanDrivenExistence	Plan driven existence. This existence object will take whatever is hardcoded in the Plan (either the MigrationPlan or the DataSourcePlan) and add those values to the HashSet. It can be left empty for use as a de-duping first-writer-wins strategy on feed data objects.
PostProcessorDescriptor
ProcessingHelper	Processing helper class. It turns out there is quite a bit of shared logic between the RowProcessor and FeedProcessor classes. This class provides services to both.
ReadyToUseSubPlan	These are the software equivalent of MRE's, i.e. 'Ready to Eat' packages of components that a datasource can consume for its use, rather than having to configure them. The idea is to achieve convention-over-configuration by establishing a set of these things in the master plan, each with its own tag, and datasourceplans can just specify which one they want to use by specifying the tag, rather than declaring everything explicitly.
ReadyToUseSubPlanDescriptor
S3FeedAccessor	S3FeedAccess implements IFeedAccess, and manages file movement for feeds deposited on S3 locations, with methods for moving, copying and deleting feeds, as well as facilitating downloading to the local file system.
ScriptMethodResolver
SqlDataSource	The SqlDataSource class can use Sql data from a table or query, and present it to the system as an IParser-based data source. It could be used for getting data from SQL Server into Redis, for example. Caveats: It assumes there is an identity column on the table or at least some sort of id or other column that is monotonically increasing. It may well work with data sources where there are 'holes' in the sequence of ('ids'), but this has not been tested. Take a look at DataMigrationPlan.xml, which shows the beginnings of an example of usage for this class.
SqlPhaseLogger	The SqlPhaseLogger class performs phase logging on behalf of the C# code. It writes to a table, specified in the PhaseLoggerDescriptor in the Plan.
SqlQueryDrivenExistence	SqlQueryDrivenExistence - this class can produce an IExistence implementation, given a query to Sql Server.
SqlQueryDrivenLookup	SqlQueryDrivenLookup is a very simple class that knows how to take the text of a sql query from its descriptor in the plan, execute it, and perform key-value lookups using the values obtained via the query. It is not for use with stored procedures.
TestScriptMethodsAndLitmusTests	This is a test script method and litmustest file; the test code will compile it on the fly, invoke the TransformMethod(s) and LitmusTest(s) in the file, and verify they work as expected.
TransformMapping
XmlDocParser	Xml document parser. Crude, dead-simple, puts-the-entire-file-in-memory parser. Do not use for feeds of any size or complexity. There are lots of features that can be added to this one for small feeds that will be added on an in-demand basis, e.g. xslt transformations as a property, change ProcessingHelper so it can use xPath fragments as value selectors, etc., blah, blah. None of that for now.