How To Synchronize Core Data with a Web Service – Part 1

Process remote service data into Core Data

Well, you have our data persisting to disk in a Property List format. But what you really want to do is to process it into Core Data. This is where you will be doing some heavy lifting and getting into the nitty gritty details. This is the part where the real magic happens!

To start off you will first need a way to retrieve the files from disk. Add -JSONDictionaryForClassWithName: in SDSyncEngine.m:

- (NSDictionary *)JSONDictionaryForClassWithName:(NSString *)className {
    NSURL *fileURL = [NSURL URLWithString:className relativeToURL:[self JSONDataRecordsDirectory]]; 
    return [NSDictionary dictionaryWithContentsOfURL:fileURL];
}

One caveat to the NSDictionary that -JSONDictionaryForClassWithName: returns is that the information you are interested in is will be in an NSArray with the key “results”. So to make things easier for processing purposes, add another method to access the data in the NSArray and spice it up a little to allow for sorting of the records by a specified key.

Add this beneath -JSONDictionaryForClassWithName: in SDSyncEngine.m:

- (NSArray *)JSONDataRecordsForClass:(NSString *)className sortedByKey:(NSString *)key {
    NSDictionary *JSONDictionary = [self JSONDictionaryForClassWithName:className];
    NSArray *records = [JSONDictionary objectForKey:@"results"];
    return [records sortedArrayUsingDescriptors:[NSArray arrayWithObject:
                                                 [NSSortDescriptor sortDescriptorWithKey:key ascending:YES]]];
}

This method calls the previous method you implemented, and returns an NSArray of all the records in the response, sorted by the specified key.

You won’t really need the JSON responses that were saved to disk much past this point, so add another method to delete them when you’re finished with them. Add the following method above -JSONDictionaryForClassWithName:

- (void)deleteJSONDataRecordsForClassWithName:(NSString *)className {
    NSURL *url = [NSURL URLWithString:className relativeToURL:[self JSONDataRecordsDirectory]];
    NSError *error = nil;
    BOOL deleted = [[NSFileManager defaultManager] removeItemAtURL:url error:&error];
    if (!deleted) {
        NSLog(@"Unable to delete JSON Records at %@, reason: %@", url, error);
    }
}

In order to translate records from JSON to NSManagedObjects, you will need a few methods. First, you will need to translate the JSON values to Objective-C properties; the method you use will vary based on the remote service you are working with. In this case, you are using Parse which has a few “special” data types. The data you’ll be concerned with here are Files and Dates. Files are returned as URLs to the file’s location, and Dates are returned in the following format:

{
  "__type": "Date",
  "iso": "2011-08-21T18:02:52.249Z"
}

Since the date is in the format of a string, you will want some methods to convert from a Parse formatted date string to an NSDate and back to an NSString. NSDateFormatter can help with this, but they are very expensive to allocate — so first add a new NSDateFormatter property that you can re-use.

Add the dateFormatter property in your private category:

@interface SDSyncEngine ()

@property (nonatomic, strong) NSMutableArray *registeredClassesToSync;
@property (nonatomic, strong) NSDateFormatter *dateFormatter;

@end

Don’t forget to synthesize it as well in your @implementation:

@synthesize dateFormatter = _dateFormatter;

And add these three methods above the #pragma mark -File Management.

- (void)initializeDateFormatter {
    if (!self.dateFormatter) {
        self.dateFormatter = [[NSDateFormatter alloc] init];
        [self.dateFormatter setDateFormat:@"yyyy-MM-dd'T'HH:mm:ss'Z'"];
        [self.dateFormatter setTimeZone:[NSTimeZone timeZoneWithName:@"GMT"]];
    }
}

- (NSDate *)dateUsingStringFromAPI:(NSString *)dateString {
    [self initializeDateFormatter];
    // NSDateFormatter does not like ISO 8601 so strip the milliseconds and timezone
    dateString = [dateString substringWithRange:NSMakeRange(0, [dateString length]-5)];

    return [self.dateFormatter dateFromString:dateString];
}

- (NSString *)dateStringForAPIUsingDate:(NSDate *)date {
    [self initializeDateFormatter];
    NSString *dateString = [self.dateFormatter stringFromDate:date];
    // remove Z
    dateString = [dateString substringWithRange:NSMakeRange(0, [dateString length]-1)];
    // add milliseconds and put Z back on
    dateString = [dateString stringByAppendingFormat:@".000Z"];

    return dateString;
}

The first method -initializeDateFormatter will initialize your dateFormatter property. The second method -dateUsingStringFromAPI: receives an NSString and returns an NSDate object. The third method -dateStringForAPIUsingDate: receives an NSDate and returns an NSString.

Take a little closer look, there, detective — the second and third methods do something a little strange. Parse uses timestamps in the ISO 8601 format which do not translate to NSDate objects very well, so you need to do some stripping and appending of the milliseconds and Z flag (used to denote the timezone). (Oh standards…there are so many wonderful ones to choose from!) :]

Next add this method below mostRecentUpdatedAtDateForEntityWithName:

- (void)setValue:(id)value forKey:(NSString *)key forManagedObject:(NSManagedObject *)managedObject {
    if ([key isEqualToString:@"createdAt"] || [key isEqualToString:@"updatedAt"]) {
        NSDate *date = [self dateUsingStringFromAPI:value];
        [managedObject setValue:date forKey:key];
    } else if ([value isKindOfClass:[NSDictionary class]]) {
        if ([value objectForKey:@"__type"]) {
            NSString *dataType = [value objectForKey:@"__type"];
            if ([dataType isEqualToString:@"Date"]) {
                NSString *dateString = [value objectForKey:@"iso"];
                NSDate *date = [self dateUsingStringFromAPI:dateString];
                [managedObject setValue:date forKey:key];
            } else if ([dataType isEqualToString:@"File"]) {
                NSString *urlString = [value objectForKey:@"url"];
                NSURL *url = [NSURL URLWithString:urlString];
                NSURLRequest *request = [NSURLRequest requestWithURL:url];
                NSURLResponse *response = nil;
                NSError *error = nil;
                NSData *dataResponse = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error];
                [managedObject setValue:dataResponse forKey:key];
            } else {
                NSLog(@"Unknown Data Type Received");
                [managedObject setValue:nil forKey:key];
            }
        }
    } else {
        [managedObject setValue:value forKey:key];
    }
}

This method accepts a value, key, and managedObject. If the key is equal to createdDate or updatedAt, you will be converting them to NSDates. If the key is an NSDictionary you will check the __type key to determine the data type Parse returned. If it is a Date, you will convert the value from an NSString to an NSDate. If it is a File, you will do a little more work since you are interested in getting the image itself!

To get the image, send off a request to download the image file. It is important to note that downloading the image data can take a considerable amount of time, so this may only work efficiently with smaller data sets. Another solution would be to fetch the image data when the record is accessed (lazy loading), but it would only be available if the user has an Internet connection at the time of lazy loading.

If the data type is anything other than a File or Date there is no way to know what to do with it so set the value to nil. In any other case you will simply pass the value and key through untouched and set them on the managedObject.

Next, add methods that create an NSManagedObject or update an NSManagedObject based on a record from the JSON response to SDSyncEngine.h:

typedef enum {
    SDObjectSynced = 0,
    SDObjectCreated,
    SDObjectDeleted,
} SDObjectSyncStatus;

Then add these two methods right above setValue:forKey:forManagedObject:

- (void)newManagedObjectWithClassName:(NSString *)className forRecord:(NSDictionary *)record {
    NSManagedObject *newManagedObject = [NSEntityDescription insertNewObjectForEntityForName:className inManagedObjectContext:[[SDCoreDataController sharedInstance] backgroundManagedObjectContext]];
    [record enumerateKeysAndObjectsUsingBlock:^(id key, id obj, BOOL *stop) {
        [self setValue:obj forKey:key forManagedObject:newManagedObject];
    }];
    [record setValue:[NSNumber numberWithInt:SDObjectSynced] forKey:@"syncStatus"];
}

- (void)updateManagedObject:(NSManagedObject *)managedObject withRecord:(NSDictionary *)record {
    [record enumerateKeysAndObjectsUsingBlock:^(id key, id obj, BOOL *stop) {
        [self setValue:obj forKey:key forManagedObject:managedObject];
    }];
}

• newManagedObjectWithClassName:forRecord: accepts a className and a record, using this information it will create a new NSManagedObject in the backgroundManagedObjectContext
• -updateManagedObject:withRecord: accepts an NSManagedObject and a record, using this information it will update the passed NSManagedObject with the record information in the backgroundManagedObjectContext

You’re heading into the home stretch! Just two more methods before you tie it all together in a method that actually processes the JSON data into Core Data. Add these methods right after -setValue:forKey:forManagedObject:

- (NSArray *)managedObjectsForClass:(NSString *)className withSyncStatus:(SDObjectSyncStatus)syncStatus {
    __block NSArray *results = nil;
    NSManagedObjectContext *managedObjectContext = [[SDCoreDataController sharedInstance] backgroundManagedObjectContext];
    NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:className];
    NSPredicate *predicate = [NSPredicate predicateWithFormat:@"syncStatus = %d", syncStatus];
    [fetchRequest setPredicate:predicate];
    [managedObjectContext performBlockAndWait:^{
        NSError *error = nil;
        results = [managedObjectContext executeFetchRequest:fetchRequest error:&error];
    }];

    return results;    
}

- (NSArray *)managedObjectsForClass:(NSString *)className sortedByKey:(NSString *)key usingArrayOfIds:(NSArray *)idArray inArrayOfIds:(BOOL)inIds {
    __block NSArray *results = nil;
    NSManagedObjectContext *managedObjectContext = [[SDCoreDataController sharedInstance] backgroundManagedObjectContext];
    NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:className];
    NSPredicate *predicate;
    if (inIds) {
        predicate = [NSPredicate predicateWithFormat:@"objectId IN %@", idArray];
    } else {
        predicate = [NSPredicate predicateWithFormat:@"NOT (objectId IN %@)", idArray];
    }

    [fetchRequest setPredicate:predicate];
    [fetchRequest setSortDescriptors:[NSArray arrayWithObject:
                                      [NSSortDescriptor sortDescriptorWithKey:@"objectId" ascending:YES]]];
    [managedObjectContext performBlockAndWait:^{
        NSError *error = nil;
        results = [managedObjectContext executeFetchRequest:fetchRequest error:&error];
    }];

    return results;
}

• -managedObjectsForClass:withSyncStatus: returns an NSArray of NSManagedObjects for the specified className where their syncStatus is set to the specified status,
• -managedObjectsForClass:sortedByKey:usingArrayOfIds:inArrayOfIds: returns an NSArray of NSManagedObjects for the specified className, sorted by key, using an array of objectIds, and you can tell the method to return NSManagedObjects whose objectIds match those in the passed array or those who do not match those in the array. You’ll read more about this method later on.

Now to put all of these methods to use!

- (void)processJSONDataRecordsIntoCoreData {
    NSManagedObjectContext *managedObjectContext = [[SDCoreDataController sharedInstance] backgroundManagedObjectContext];
    //
    // Iterate over all registered classes to sync
    //
    for (NSString *className in self.registeredClassesToSync) {
        if (![self initialSyncComplete]) { // import all downloaded data to Core Data for initial sync
            //
            // If this is the initial sync then the logic is pretty simple, you will fetch the JSON data from disk 
            // for the class of the current iteration and create new NSManagedObjects for each record
            //
            NSDictionary *JSONDictionary = [self JSONDictionaryForClassWithName:className];
            NSArray *records = [JSONDictionary objectForKey:@"results"];
            for (NSDictionary *record in records) {
                [self newManagedObjectWithClassName:className forRecord:record];
            }
        } else {
            //
            // Otherwise you need to do some more logic to determine if the record is new or has been updated. 
            // First get the downloaded records from the JSON response, verify there is at least one object in 
            // the data, and then fetch all records stored in Core Data whose objectId matches those from the JSON response.
            //
            NSArray *downloadedRecords = [self JSONDataRecordsForClass:className sortedByKey:@"objectId"];
            if ([downloadedRecords lastObject]) {
                //
                // Now you have a set of objects from the remote service and all of the matching objects 
                // (based on objectId) from your Core Data store. Iterate over all of the downloaded records 
                // from the remote service.
                //
                NSArray *storedRecords = [self managedObjectsForClass:className sortedByKey:@"objectId" usingArrayOfIds:[downloadedRecords valueForKey:@"objectId"] inArrayOfIds:YES];
                int currentIndex = 0;
                // 
                // If the number of records in your Core Data store is less than the currentIndex, you know that 
                // you have a potential match between the downloaded records and stored records because you sorted 
                // both lists by objectId, this means that an update has come in from the remote service
                //
                for (NSDictionary *record in downloadedRecords) {
                    NSManagedObject *storedManagedObject = nil;
                    
                    // Make sure we don't access an index that is out of bounds as we are iterating over both collections together
                    if ([storedRecords count] > currentIndex) {
                        storedManagedObject = [storedRecords objectAtIndex:currentIndex];
                    }

                    if ([[storedManagedObject valueForKey:@"objectId"] isEqualToString:[record valueForKey:@"objectId"]]) {
                        //
                        // Do a quick spot check to validate the objectIds in fact do match, if they do update the stored 
                        // object with the values received from the remote service
                        //
                        [self updateManagedObject:[storedRecords objectAtIndex:currentIndex] withRecord:record];
                    } else {
                        //
                        // Otherwise you have a new object coming in from your remote service so create a new 
                        // NSManagedObject to represent this remote object locally
                        //
                        [self newManagedObjectWithClassName:className forRecord:record];
                    }
                    currentIndex++;
                }
            }
        }
        //
        // Once all NSManagedObjects are created in your context you can save the context to persist the objects 
        // to your persistent store. In this case though you used an NSManagedObjectContext who has a parent context 
        // so all changes will be pushed to the parent context
        //
        [managedObjectContext performBlockAndWait:^{
            NSError *error = nil;
            if (![managedObjectContext save:&error]) {
                NSLog(@"Unable to save context for class %@", className);
            }
        }];

        //
        // You are now done with the downloaded JSON responses so you can delete them to clean up after yourself, 
        // then call your -executeSyncCompletedOperations to save off your master context and set the 
        // syncInProgress flag to NO
        //
        [self deleteJSONDataRecordsForClassWithName:className];
        [self executeSyncCompletedOperations];
    }
}

This method must be called where Comment 2 sits as a placeholder in the downloadDataForRegisteredObjects method. When the HTTP request operations are completed, returned data is written to Core Data, and your app can then access them as required.

This is it, kids! This is the final downloadDataForRegisteredObjects method!

- (void)downloadDataForRegisteredObjects:(BOOL)useUpdatedAtDate {
    NSMutableArray *operations = [NSMutableArray array];
    
    for (NSString *className in self.registeredClassesToSync) {
        NSDate *mostRecentUpdatedDate = nil;
        if (useUpdatedAtDate) {
            mostRecentUpdatedDate = [self mostRecentUpdatedAtDateForEntityWithName:className];
        }
        NSMutableURLRequest *request = [[SDAFParseAPIClient sharedClient]
                                        GETRequestForAllRecordsOfClass:className
                                        updatedAfterDate:mostRecentUpdatedDate];
        AFHTTPRequestOperation *operation = [[SDAFParseAPIClient sharedClient] HTTPRequestOperationWithRequest:request success:^(AFHTTPRequestOperation *operation, id responseObject) {
            if ([responseObject isKindOfClass:[NSDictionary class]]) {
                [self writeJSONResponse:responseObject toDiskForClassWithName:className];
            }
        } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
            NSLog(@"Request for class %@ failed with error: %@", className, error);
        }];
        
        [operations addObject:operation];
    }
    
    [[SDAFParseAPIClient sharedClient] enqueueBatchOfHTTPRequestOperations:operations progressBlock:^(NSUInteger numberOfCompletedOperations, NSUInteger totalNumberOfOperations) {
        
    } completionBlock:^(NSArray *operations) {

            [self processJSONDataRecordsIntoCoreData];
    }];
}

Last thing: to see the effect, go to SDDateTableViewController.m and update -viewDidAppear: and -viewDidDisappear: to register for the sync complete notification and reload the table:

- (void)viewDidAppear:(BOOL)animated {
    [super viewDidAppear:animated];

    [[NSNotificationCenter defaultCenter] addObserverForName:@"SDSyncEngineSyncCompleted" object:nil queue:nil usingBlock:^(NSNotification *note) {
        [self loadRecordsFromCoreData];
        [self.tableView reloadData];
    }];
}

- (void)viewDidDisappear:(BOOL)animated {
    [super viewDidDisappear:animated];
    [[NSNotificationCenter defaultCenter] removeObserver:self name:@"SDSyncEngineSyncCompleted" object:nil];
}

Run the project and see your records created in Parse come in to your App!