KV Basics

Let’s talk a bit here about the basics of how my key-value system works. It’ll take a few articles to fill in all the details, but I’ll try to keep each section self-sufficient. Each one may leave you wanting more details, and I plan to provide them. I’ll also watch the comments on the articles to see if they help me improve existing articles or write new ones. It’ll help if I get comments while I’m still writing.

Inside SL

Inside SL, the KV system works by sending an HTTP request to the Amazon Web Services (AWS) API, and getting a reply back. For architectural reasons, and security reasons, I’ve got a little core script with a link_message event and an http_response event, and the main library initiates a request by linking in, and gets a response when the core script links back. I’ll show the details later, but it looks kind of like this:

link_message(integer sender_num, integer num, string query, key body) {
...
// API is the http address of the KV API
string url = API + "/read?pk=bill&sk=001&table=janskvtable";
key k = llHTTPRequest(url, [], "");
KV_List += [ k, num-100];
}

http_response(key request_id, integer status, list metadata, string body) {
integer index = llListFindList(KV_List, [request_id]);
if ( index == -1 ) {llOwnerSay("not found"); return; }
llMessageLinked(LINK_THIS, llList2Integer(KV_List, index+1), body, "");
KV_List = llDeleteSubList(KV_List, index, index+1);
}

We’ll leave the details of how the link messages work until later, but basically num tells the script which action is needed (read, write, and so on), the script wraps up the right URL, issues the request, and saves the request key and the input number minus 100 in a list. When the response comes in, it looks up the list entry, and calls back on the number provided. So when you come in asking for a read request with whatever that number is, maybe 506, you’ll be called back on 406 when the request completes.

The information returned will be expressed in JSON, and we’ll look at the details of that later. The record returned here would look like this:

{"Item":{"value":{"S":"this is bill's first record"},
"vt_sk":{"S":"001"},"vt_pk":{"S":"bill"}}}

The construction of the HTTP request is of course dynamic but the example given is typical. There’s the main AWS API, slash, a request word like read, followed by the query string, a list of pairs of keyword and value, like pk=bill&sk=001. In this case, pk refers to the primary key of the record we want, and sk to the secondary or sort key. Each record in our database has two keys, primary and secondary. That would let bill have several records. In a budget database, they might be bill, rent, bill,carpayment, and so on.

So, inside SL, at base, it’s simple enough. We link to core from the main library, with some parameters. Core makes an HTTP request, and when the response comes back, we link back to the main library. The main library, as we’ll see, calls back to the code you provide, to do whatever you want with the record you read and so on.

On the AWS side

On the AWS side, it’s not much more complicated, as we’ll see, and the setup is simple.

You’re defining what Amazon calls an API. It has a somewhat user-friendly name: the one I’m working with now is named key-aux-value-sl. And it has a generated real URL, like

https://tz43xy4ig9.execute-api.us-east-1.amazonaws.com/vt-kv-1.

(Don’t bother to try that one, I just typed in random characters in that first bit. Your real API wants to be a bit confidential, since people could at least attempt to access it even if it has authentication on it. That’s part of why my scripts will separate out the core, which will not be readable by users.)

The API provides “Resources”. Mine presently provides delete, query, read, and write. There will be a few more. Each Resource can implement one or more HTTP method. Presently mine only implement one method. The first three implement GET, and the write resource implements PUT.

When you look at a given Resource, you see a picture like this:

 
You see “Client” on the left, “DynamoDB” on the right, and two inbound boxes and two outbound boxes, representing how requests get from the API to and from DynamoDB. Each of those boxes can be customized to control what goes in and out. At this writing, I’ve only needed to customize the “Integration Request”, which maps from the input query line into something that DynamoDB understands. The mapping template in read looks like this:

{
  "TableName": "$input.params('table')",
  "Key": {
    "vt_pk": { "S": "$input.params('pk')" }
    #if( $input.params('sk') != "" )
    ,"vt_sk": { "S": "$input.params('sk')" }
    #end
    }
  }
}

This is cryptic and I often found it difficult to find an example I could hammer into shape for my needs. But what we’re saying here is to look at the input parameters, the ones on the query line show above, and plug them into this template, to make the real request. The real request would look something like this after plugging:

{
  "TableName": "janskvtable",
  "Key": {
    "vt_pk": "bill" 
    ,"vt_sk": "001"
    }
  }
}

(Wow! I just noticed, plugging those values in, that the brackets don’t seem to balance in that expression. AWS doesn’t complain about it: the query works fine. I just tested it above. I’ll have to look into that. It must be somewhat forgiving, which has not been my usual experience. Anyway, that’s what’s in there right now, and it does work. (P.S. I changed it on AWS to have balanced brackets and it still works!))

The fields in my DynamoDB table are named vt_pk, vt_sk, and value. In the URL I used pk and sk for convenience, and the translation is done here.

The operation when the API Resource is called is to go through the Request boxes, run DynamoDB to get the result, and go back through the Response boxes to return the result. As I mentioned, right now, I’m only doing the one translation, to plug in the query parameters, and I’m passing the DynamoDB results back out. That may need to be improved, to deal with errors. We’ll see as time goes on.

Summing up

So that’s the big picture. Our Second Life script provides some key values and a request type, and links to the core script. The core script makes a fairly simple HTTP request to the API, saying what it wants to do and what the parameters are. The API plugs the query parameters into a template that DynamoDB understands, makes the request, and passes the result back to the core script. The core script passes the results back via a link message, and our script does whatever it wants with the information from the database.

• User Script: Provide key values and request type;
• User Script: Link to core;
• Core Script: Make HTTP Request to API;
• AWS API: Format request for DynamoDB;
• AWS API: Forward to DynamoDB;
• DynamoDB: Read or write data;
• DynamoDB: Return data to API;
• AWS API: Respond to HTTP Request;
• Core Script: Receive HTTP Response;
• Core Script: Link to user script;
• User Script: Process the information.

Nothing to it! It all happens inside a few simple calls. If all goes well. We’ll look at some more details next time. Stay tuned!