Sunday, August 13, 2017

[kyorohiro/cgame] Object Oriented At Clang

I had wanted to use object oriented  programming frame in Clang.
I created core module in kyorohiro/cgame.

Ex
https://github.com/kyorohiro/cgame/blob/master/core/cobject.h


kyorohiro/cgame's core has following.
 - Inheritance mechanism
 - delegation
 - basic collection
 - event delegation



Saturday, August 12, 2017

[kyorohiro/cgame] Started to develop clang game library

I had started to develop clang game library.


This project purpose is following 
- kyorohiro get 3d game know-how.
- know-how is use in c program book
- create clang game library for emscripten

In this project, use C language not to C++ language.

What do you think is the programming language to be first learned.
What if I learn c language at first time. It is difficult to grasp the whole picture.

As I can understand program language grammer. I can not create application.
It is because you do not understand the mechanism of the program that actually works in the world

I assumed that We must to learn from goal.  We should start by touching the code of the program that actually works.


then , I had started this project.




Monday, August 17, 2015

HetimaNet library support dart:io sockets.

HetimaNet library for dart developer support dart:io socket.  HetimaNet had supported chrome application extension sockets only.  


HetimaNet have following function
  • UDP
  • TCP
  • Http Server
  • Http Client
  • UPnP Portmap 



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import 'package:hetimanet/hetimanet.dart';
import 'package:hetimanet/hetimanet_dartio.dart';
//
//
main()  async {
  HetiSocketBuilderDartIO builder = new HetiSocketBuilderDartIO(); 
  UpnpPortMapHelper helper = new UpnpPortMapHelper(builder, "test");
  //
  // get network interface
  List<HetiNetworkInterface> interfaces = await builder.getNetworkInterfaces();
  for (HetiNetworkInterface i in interfaces) {
    print("<ni>${i.address} ${i.prefixLength} ${i.name}");
  }
  //
  // portmapping 
  try {
    StartGetExternalIp exip = await helper.startGetExternalIp(reuseRouter: true);
    print("<exip> ${exip.externalIp}");
  } catch (e) {
    print("<exip ERROR> ${e}");
  }
  //
  // get local ip
  try {
    StartGetLocalIPResult loip = await helper.startGetLocalIp();
    for(HetiNetworkInterface i in loip.networkInterface) {
      print("<glip> ${i.address} ${i.name}");      
    }
  } catch (e) {
    print("<glip ERROR> ${e}");
  }
  //
  // start portmap
  try {
    StartPortMapResult sp = await helper.startPortMap();
    print("<add> ${sp}");
  } catch (e) {
    print("<add ERROR> ${e}");
  }
  //
  // end portmap
  try {
    DeleteAllPortMapResult ep = await helper.deletePortMapFromAppIdDesc();
    print("<del> ${ep}");
  } catch (e) {
    print("<del ERROR> ${e}");
  }
}

Source

https://github.com/kyorohiro/dart_hetimanet
https://github.com/kyorohiro/dart_hetimanet/tree/master/example/HetimaPortMap

Port map chrome app

- Chrome Store

https://chrome.google.com/webstore/detail/hetimaportmap/naifildeohmcocnmibaampijofhcohif

--------------
kyorohiro work



Friday, August 14, 2015

Let's coding mainline dht.(6) End

Can find value to use get_peers message


  • use get_peers message to find value
  • use announce_peer message  to save value in p2p network


We could create p2p network in previous section. I explain to save value in p2p network.

get_peers message and find_node are almost similar.


MainLine DHT use get_peers message to set value in p2p network. get_peers message have two role.

  • when receive get_peers query, if knode have request value, knode  send response message about value.
  • if knode dont have value, knode send reposese message about close peer from kid.

Knode repose to send get_peers message, then, knode can find close peer, and value.

Those system and find_nodes is almost simillar.

  •  introduce close peer form KID
  •  repeate to send meesage 




save data to use announce_peer message

When Knode find k-number of close knode to repeat sending get_peers message. Knode send announce_peer message to save value.

Knode send announce_peer message into k-number node.
A robustness rises by requesting value into multiple node.


  • It is easy to find valie
  • It is safe to disconnect node
  • It's possible to disperse a load.


message struct


```
arguments:
{
  "t":"aa",
  "y":"q",
  "q":"get_peers",
  "a": {
    "id" : "<querying nodes id>",
    "info_hash" : "<20-byte infohash of target torrent>"
  }
}

response: have value
{
  "id" : "<queried nodes id>",
  "token" :"<opaque write token>",
  "values" :
  ["<peer 1 info string>", "<peer 2 info string>"]
}

response: have node info
{
 "t":"aa",
 "y":"r",
 "r": {
  "id" : "<queried nodes id>",
  "token" :"<opaque write token>",
  "nodes" : "<compact node info>"
  }
}
```

"token" and "values" is  appeared for the first time int this blog.
"token" is byte date that responser define this byte data and byte data length. This value ise used to send announce_peers query.
"value" is list that contain 6bytes data[<IP 4bute> ,<Port 2byte>]. this values is peer ip and port that's peer can share file to use torrent client bep003.

ref http://www.bittorrent.org/beps/bep_0005.html


About Implements 


get_peers implements is almost similar as find_node. We can code get_peers to use find_node implements 's know-how.



Test DHT


  •  test dht

You'll make them move actually. I explain mainline dht sample to use hetimatorrent library.

find value from special kid  int p2p network.


```

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
main() {
  KNode node = new KNode(new HetiSocketBuilderChrome(), verbose: true);

  //
  node.start(ip: "0.0.0.0", port: 28080).then((_) {
    node.onGetPeerValue.listen((KGetPeerValue v) {
      print("---onGetPeerValue ${v.ipAsString} ${v.port} ${v.infoHashAsString} ");
    });
  });

  // initial node
  String initailNodeIp = "0.0.0.0";
  int initailNodePort = 38080;
  node.addBootNode(initailNodeIp, initailNodePort);
  
  // search target
  List<int> infoHash = new List.filled(20, 4);
  node.startSearchValue(new KId(infoHash), 18080, getPeerOnly: true);
  
  new Future.delayed(new Duration(minutes:30)).then((_){
    print(node.rootingtable.toInfo());
    return node.stop();
  });

}
```


Sample app with gui.

https://github.com/kyorohiro/dart_hetimatorrent/tree/master/example/TorrentDHT




Ref


http://www.bittorrent.org/beps/bep_0005.html
http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf


PS

GitBook nazenani Torrent for japanese
- https://www.gitbook.com/book/kyorohiro/doc_hetimatorrent/details
hetima torrent library
- https://github.com/kyorohiro/dart_hetimatorrent
- https://github.com/kyorohiro/dart_hetimatorrent/tree/master/example/TorrentDHT


-------
Kyorohiro work

http://kyorohiro.strikingly.com

Thursday, August 13, 2015

Let's coding mainline dht.(5) Implements FindNodes

Implements FindNodes

(1) Add findnode function in RootingTable.


First. I add findnode function in rootingtable class.  This function can search close peer info from  target KID to sort KIDs .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class KRootingTable {
  ...
  ...
  ...
  List<KPeerInfo> findNode(KId id) {
    List<KPeerInfo> ids = [];
    for (KBucket b in _kBuckets) {
      for (KPeerInfo i in b.iterable) {
        ids.add(i);
      }
    }
    ids.sort((KPeerInfo a, KPeerInfo b) {
      return a.id.xor(id).compareTo(b.id.xor(id));
    });
    List<KPeerInfo> ret = [];
     for (KPeerInfo p in ids) {
      ret.add(p);
       if (ret.length >= _kBucketSize) {
         return ret;
      }
    }
    return ret;
  }
  ...
  ...
}

(1) KNode have UDP server function

Knode use udp socket to communication between node and node. and use bencode. This term add binding to use udp server and parse message.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
class KNode {
  bool _isStart = false;
  bool get isStart => _isStart;
  HetiSocketBuilder _socketBuilder = null;
  HetiUdpSocket _udpSocket = null;

  KNode(HetiSocketBuilder socketBuilder) {
    this._socketBuilder = socketBuilder;
  }
 
  Future start({String ip: "0.0.0.0", int port: 28080}) async {
    (_isStart != false ? throw "already started" : 0);
    _udpSocket = this._socketBuilder.createUdpClient();
    return _udpSocket.bind(ip, port, multicast: true).then((int v) {
      _udpSocket.onReceive().listen((HetiReceiveUdpInfo info) {
        KrpcMessage.decode(info.data, this).then((KrpcMessage message) {
          onReceiveMessage(info, message);
        });
      });
      _isStart = true;
    });
  }

  Future stop() async {
    if (_isStart == false || _udpSocket == null) {
      return null;
    }
    return _udpSocket.close().whenComplete(() {
      _isStart = false;
      _ai.stop(this);
    });
  }
}

(2) KNode have Krpc Message parse function.


MainLine DHT use KRPC protocol with bencode. this term create to prase krpc message . but We had already create bencod barser. It is easy.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
class KrpcMessage {
  KrpcMessage.fromMap(Map map) {
    _messageAsMap = map;
  }

  static Future<KrpcMessage> decode(List<int> data) async {
    Map<String, Object> messageAsMap = null;
    try {
      Object v = Bencode.decode(data);
      messageAsMap = v;
    } catch (e) {
      throw {};
    }
    return  new KrpcMessage.fromMap(messageAsMap);
  }
}


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
class KrpcMessage {
...
...
  List<int> get transactionId => _messageAsMap["t"]);
  String get transactionIdAsString => UTF8.decode(transactionId);

  //
  List<int> get messageType => _messageAsMap["y"];
  String get messageTypeAsString => UTF8.decode(messageType);

  //
  List<int> get query => _messageAsMap["q"];
...
...
}
````


(3) KNode have seding message function


```

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
class KNode {
 ..
 ..
  sendMessage(KrpcMessage message, String ip, int port) {
      return _udpSocket.send(message.messageAsBencode, ip, port);
  }
 ..
}

class FindNode {

  static int queryID = 0;

  static KrpcMessage createQuery(List<int> queryingNodesId, List<int> targetNodeId) {
    List<int> transactionId = UTF8.encode("fi${queryID++}");
    return new KrpcMessage.fromMap({"a": {"id": queryingNodesId, "target": targetNodeId}, "q": "find_node", "t": transactionId, "y": "q"});
  }

  static KrpcMessage createResponse(List<int> compactNodeInfo, List<int> queryingNodesId, List<int> transactionId) {
    return new KrpcMessage.fromMap({"r": {"id": queryingNodesId, "nodes": compactNodeInfo}, "t": transactionId, "y": "r"});
  }
}
```

When send message, KNode use binded UDPSocket. Received message node can know port and ip  to use  socket to receive message.

(4) KNode have join network function


KNode send findnode query that k-number close peer in rootingTable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
class KNodeWorkFindNode {
  ...
  ...
  updateP2PNetworkWithoutClear(KNode node) {
    node.rootingtable.findNode(node.nodeId).then((List<KPeerInfo> infos) {
      for (KPeerInfo info in infos) {
        if (!_findNodesInfo.rawsequential.contains(info)) {
          _findNodesInfo.addLast(info);
          node.sendFindNodeQuery(info.ipAsString, info.port, node.nodeId.value).catchError((_) {});
        }
      }
    });
  }
  ...
  ...
}

when receive response, then KNode send findnode query again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
class KNodeWorkFindNode {
  ...
  ...
  onReceiveQuery(KNode node, HetiReceiveUdpInfo info, KrpcMessage query) {
    if (query.queryAsString == KrpcMessage.QUERY_FIND_NODE) {
      KrpcFindNode findNode = query.toFindNode();
      return node.rootingtable.findNode(findNode.targetAsKId).then((List<KPeerInfo> infos) {
        return node.sendFindNodeResponse(info.remoteAddress, info.remotePort, query.transactionId, KPeerInfo.toCompactNodeInfos(infos)).catchError((_) {});
      });
    }
    node.rootingtable.update(new KPeerInfo(info.remoteAddress, info.remotePort, query.nodeIdAsKId));
    updateP2PNetworkWithoutClear(node);
  }
  ...
  ..
}

and If you were a certain period of time, to access it again


1
2
3
4
5
6
7
8
class KNodeWorkFindNode {
  ...
  ...

  onTicket(KNode node) {
    _findNodesInfo.clear();
    updateP2PNetworkWithoutClear(node);
  }


(5) When receive FindeNode query, KNode send findnode response


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
class KNodeWorkFindNode {
  ...
  ...
  onReceiveResponse(KNode node, HetiReceiveUdpInfo info, KrpcMessage response) {
    if (response.queryFromTransactionId == KrpcMessage.QUERY_FIND_NODE) {
      KrpcFindNode findNode = response.toFindNode();
      for (KPeerInfo info in findNode.compactNodeInfoAsKPeerInfo) {
        node.rootingtable.update(info);
      }
    }
    node.rootingtable.update(new KPeerInfo(info.remoteAddress, info.remotePort, response.nodeIdAsKId));
    updateP2PNetworkWithoutClear(node);
  }
  ..
}


Next

About GetPeers

Ref

http://www.bittorrent.org/beps/bep_0005.html
http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf

PS

- GitBook Nazenani Torrent (for japanese)
https://www.gitbook.com/book/kyorohiro/doc_hetimatorrent/details

-source
  https://github.com/kyorohiro/dart_hetimatorrent
 https://github.com/kyorohiro/dart_hetimatorrent/tree/master/example/TorrentDHT

-------
Kyorohiro work



Tuesday, August 11, 2015

About Nazenani Torrent

I am writing ebook about The know-how gained during the Torrent Client development.

- Nazenani Torrent (for japanese only now)
https://www.gitbook.com/book/kyorohiro/doc_hetimatorrent/details

-------
Kyorohiro work

Let's coding mainline dht.(4) build p2p network at findNodes query

My English skill is poor. If I make mistakes in my English,please pardon me.

Build p2p betwork at FindNode query

  • Build p2p network to use only FindNodes query
  • Update routingtable to receive a response.
I explained about kBucket Rooting table in previous section.  In this section, We build p2p network to update rooting table.


Could build network by findnodes qurey

Mainline DHT(Kademlia) use UDP socket to communicatio other peer.
P2P network is building to use only findnodes qurey.

Could find close distance peer from the specified KID

We use FindNode query,  We could find close distance peer from the specified KID.

findnodes query and response have following structure.
 arguments:  
 {  
  "t":"aa",  
  "y":"q",  
  "q":"find_node",  
  "a":{  
  "id" : "<nodes id>",  
  "target" : "<id of target node>"  
  }  
 }  
 response:  
 {  
  "t":"aa",  
  "y":"r",  
  "r":{  
  "id" : "<nodes id>",  
  "nodes" : "<compact node info>"  
  }  
 }  

"t" is specified string. that any sender can be determined.this value use when receive response.

"id" is sender peer's peer Id
"target" findnode querry return peers that the distance is close.
"nodes" is 26byte byte data list. this data have 20byte Peer ID, and IP, and port,

Join/Build p2p network.

It is updating rooting table to use findnodes query that "id" is own peer id.
repeat to request findnodes query until k-number of close peer is no longer update.

following table is a peer 's rooting table on simulator use 1000 peer.
Rooting Table indexnumner of Peercomment
01own
1471..
1541..
1554..
1564..
1576..
1588..
1598..
1608..

Become same result If try this in torrent 's dht network .

147 is very close node.

Ref


- GitBook Nazenani Torrent for japanese


- kyorohiro work torrent library and dht demo


-------
Kyorohiro work