一般用掃瞄器物件(Scanner)時,若依下法, 在讀數字後,再讀文字,會有讀不到文字情況,Scanner sc = new Scanner(System.in); int i = sc.nextInt(); // 讀數字,換行字元未讀走 String s = sc.nextLine(); // 讀換行前字串,會收到空字串這時,建議改用如下寫法,Scanner sc = new Scanner(System.in); int i = sc.nextInt(); // 讀數字,換行字元未讀走 String s = sc.nextLine(); // 故意用nextLine讀走下一換行前字串及換行字元 String s = sc.nextLine(); // 再用nextLine重新讀下一換行前字串及換行字元或Scanner sc = new Scanner(System.in); int i = sc.nextInt(); // 讀數字,換行字元未讀走 String s = sc.next(); // 讀下一個空格隔開前文字,這樣非空格文字仍可讀到如下測試範例 Test2.java 可自行試看看. ---- Test2.java -----------/* Test2.java >javac Test2.java >java Test2 input i=123 input s= input s2=456 i=123, s=, s2=456, end */ import java.util.*; public class Test2 { public static void main(String[] arg) { Scanner sc = new Scanner(System.in); System.err.printf("input i="); int i = sc.nextInt(); System.err.printf("input s="); String s = sc.nextLine(); // 前面有讀數字,後面文字會讀不到,只收到空字串 System.err.printf("\ninput s2="); String s2 = sc.nextLine(); System.err.printf("i=%d, s=%s, s2=%s, end\n",i,s, s2); } }
scanner.next and nextLine
summary of time complexity for common data structures
常見資料結構之平均時間複雜度
=======================
運算之平均時間複雜度 新增 刪除 查詢
--限特定位置增刪查詢之線性容器--
陣列堆疊(尾入尾出) O(1) O(1) O(1)
鏈結堆疊(頭入頭出) O(1) O(1) O(1)
陣列佇列(環狀尾入頭出) O(1) O(1) O(1)
鏈結佇列(尾入頭出) O(1) O(1) O(1)
陣列雙邊佇列(環狀頭尾出入) O(1) O(1) O(1)
鏈結雙邊佇列(雙向頭尾出入) O(1) O(1) O(1)
--線性容器--
無序陣列清單 O(1) O(n) O(n)
無序鏈結清單 O(1) O(n) O(n)
有序陣列清單 O(n) O(n) O(log(n))
有序鏈結清單 O(n) O(n) O(n)
雜湊表 O(1) O(1) O(1)
--非線性容器--
二分搜索樹 O(log(n)) O(log(n)) O(log(n))
跳躍鏈結清單 O(log(n)) O(log(n)) O(log(n))
堆積(限頭才能刪除,查詢) O(log(n)) O(log(n)) O(1)
參考:
1.carrano-pie-05-data abstraction & problem solving with java
2.weiss-pie-04-data structures & problem solving using java
3.budd-awl-00-classic data structures in java
java container (jdk1.5)
java容器(since jdk1.5版) Version 0.2b, 2006/10/26
==================
各容器介面摘要(功能定義):
Collection 收藏: 放個別物件, 可重複放,無位置概念,無排序
Queue 佇列: 放個別物件, 可重複放,頭位置概念,部份排序
List 清單: 放個別物件, 可重複放,任意位置 ,無排序
Set 集合: 放個別物件,不可重複放,無位置概念,無排序
SortedSet 有序集合: 放個別物件,不可重複放,有頭尾概念,有排序
Map 映射: 放成對物件,不可重複放,無位置概念,無排序
SortedMap 有序映射: 放成對物件,不可重複放,有頭尾概念,有排序
各容器類別摘要(實作各介面的實體類別):
Collection: Vector,Stack,ArrayList,ArrayBlockingQueue,
LinkedList,PriorityQueue,HashSet,TreeSet
Queue: ArrayBlockingQueue,LinkedList,PriorityQueue
List: Vector,Stack,ArrayList,LinkedList
Set: HashSet,TreeSet
SortedSet: TreeSet
Map: HashMap,TreeMap,HashTable,Properties
SortedMap: TreeMap
--
Vector 向量,Stack 堆疊,ArrayList 陣列清單,LinkedList 鏈結清單,
ArrayBlockingQueue 陣列等候佇列,PriorityQueue 順位佇列,
HashSet 雜湊集合,TreeSet 樹狀集合,
HashMap 雜湊映射,TreeMap 樹狀映射,
HashTable 雜湊表,Properties 屬性表
容器演算法:
Collections.binarySearch/min/max/indexOfSublist/lastIndexOfSublist,
fill/copy/swap/shuffle/sort/reverse/rotate,
list/nCopies/enumeration/reverseOrder,
singleton/singletonList/singletonMap,
synchronizedCollection/synchronizedSet/synchronizedList,
synchronizedMap/synchronizedSortedSet/synchronizedSortedMap,
unmodifiableCollection/unmodifiableSet/unmodifiableList,
unmodifiableMap/unmodifiableSortedSet/unmodifiableSortedMap,
Arrays.asList/binarySearch/equals/fill/sort,
容器一般性功能須求:
新增容器,刪除容器,查詢容器元素個數,列舉容器元素,
新增元素,刪除元素,修改元素,查詢元素,擷取元素
各容器類別提供的方法:
Vector >AbstractList >AbstractCollection, 存取有同步,有元素概念,容量無限
*add/addElement/insertElementAt/addAll,
*remove/removeElement/removeElementAt,
clear/removeAllElements/removeAll/retainAll/removeRange,
*get/set/firstElement/lastElement/elementAt/setElementAt,
*indexOf/lastIndexOf,
subList/toArray/copyInto,
isEmpty/contains/containsAll/equals/hasCode,
*size/setSize/trimToSize/capacity/ensureCapacity,
toString/clone,
--
iterator/listIterartor
Stack >Vector >AbstractList >AbstractCollection, 存取有同步
*empty/push/peek/pop/search,
--
add/addElement/insertElementAt/addAll,
remove/removeElement/removeElementAt,
clear/removeAllElements/removeAll/retainAll/removeRange,
get/set/firstElement/lastElement/elementAt/setElementAt,
indexOf/lastIndexOf,
subList/toArray/copyInto,
isEmpty/equals/contains/containsAll/hasCode,
size/setSize/trimToSize/capacity/ensureCapacity,
toString/clone,
--
iterator/listIterartor
ArrayList >AbstractList >AbstractCollection, 存取無同步,容量無限
*add/addAll,
*remove/removeRange/clear,
*get/set,
*indexOf/lastIndexOf,
*isEmpty/contains,
size/trimToSize/ensureCapacity,
clone/toArray,
--
iterator/listIterartor/equals/hasCode,
--
containsAll/removeAll/retainAll/toString
ArrayBlockingQueue >AbstractQueue >AbstractCollection 存取有同步,容量有限制
clear/contains/drainTo/iterator/
offer/peek/poll/put/remove/take
remainingCapacity/size/
toArray/toString
--
add/addAll/element/remove
--
containsAll/isEmpty/removeAll/retainAll
PriorityQueue >AbstractQueue >AbstractCollection 存取無同步,容量無限
add/clear/offer/peek/poll/remove
comparator/iterator/size
--
addAll/element/remove
--
contains/containsAll/isEmpty/removeAll/retainAll
toArray/toString
LinkedList >AbstractSequentialList >AbstractList
>AbstractCollection, 存取無同步,有頭尾概念,容量無限
*add/addAll/addFirst/addLast,
*remove/removeFirst/removeLast/clear,
*get/set/getFirst/getLast,
*contains/indexOf/lastIndexOf,
size/listIterator/clone/toArray,
--
iterator
--
equals/hashCode/removeRange/subList
containsAll/isEmpty/removeAll/retainAll/toString
HashSet >AbstractSet >AbstractCollection, 存取無同步,無順序概念
*add/remove/clear,
*contains/isEmpty/size,
*iterator/clone,
--
equals/hashCode/removeAll
--
addAll/containsAll/retainAll/toArray/toString
TreeSet >AbstractSet >AbstractCollection, 存取無同步,有順序概念
*add/addAll/remove/clear,
*contains/isEmpty/size,
*iterator/clone/comparator,
*first/last,
*headSet/tailSet/subSet,
--
equals/hashCode/removeAll
--
containsAll/retainAll/toArray/toString
HashMap >AbstractMap, 存取無同步,存放成對物件,無順序概念
*get/put/putAll/remove/clear,
*isEmpty/containsKey/containsValue,
values/keySet/entrySet,
size/clone
--
equals/hashCode/toString
TreeMap >AbstractMap, 存取無同步,存放成對物件,有順序概念
*get/put/putAll/remove/clear,
*containsKey/containsValue,
*headMap,tailMap,subMap,
values/keySet/entrySet,
firstKey/lastKey,
size/clone/comparator,
--
equals/hashCode/isEmpty/toString
HashTable >Dictionaries, 存取有同步,存放成對鍵值組
*get/put/putAll/remove/clear,
*isEmpty/containsKey/containsValue/contains,
keys/elements/values/keySet/entrySet,
size/clone/equals/hasCode/rehash/toString
--
get/put/remove,
elements/keys,
size/isEmpty,
Properties >HashTable >Dictionaries, 存取有同步,存放屬性鍵值組
*getProperty/setProperty,
*load/store/list,
*propertyNames,
--
*get/put/putAll/remove/clear,
*isEmpty/containsKey/containsValue/contains,
keys/elements/values/keySet/entrySet,
size/clone/equals/hasCode/rehash/toString
各容器介面提供的方法:
Collection
add/allAll/remove/removeAll/retainAll/clear,
isEmpty/contains/containsAll/equals/hasCode/size,
iterator/toArray,
Queue >Collection 多了頭位置概念
offer/
poll/remove/
peek/element/
--
add/allAll/remove/removeAll/retainAll/clear,
isEmpty/contains/containsAll/equals/hasCode/size,
iterator/toArray,
List >Collection 多了任意位置概念
add/allAll/remove,
get/set,
indexOf/lastIndexOf,
listIterator/subList,
--
add/allAll/remove/removeAll/retainAll/clear,
isEmpty/contains/containsAll/equals/hasCode/size,
iterator/toArray,
Set >Collection 多了不重複存放概念
add/allAll/remove/removeAll/retainAll/clear,
isEmpty/contains/containsAll/equals/hasCode/size,
iterator/toArray,
SortedSet >Set >Collection 多了順序概念
first/last,
headSet/tailSet/subSet,
comparator,
--
add/allAll/remove/removeAll/retainAll/clear,
isEmpty/contains/containsAll/equals/hasCode/size,
iterator/toArray,
Map
get/put/putAll/remove/clear,
isEmpty/containsKey/containsValue,
values/keySet/entrySet,
size/equals/hasCode,
SortedMap >Map
firstKey/lastKey,
headMap/tailMap/subMap,
comparator,
--
get/put/putAll/remove/clear,
isEmpty/containsKey/containsValue,
values/keySet/entrySet,
size/equals/hasCode,
迭代列舉介面提供的方法:
Enumeration 列舉介面,不能刪除列舉物件,只能往後列舉
hasMoreElements/nextElement,
Iterator 迭代介面,可刪除列舉物件,只能往後列舉
hasNext/next/remove,
ListIterator 清單迭代介面,可往前後列舉,及增刪改物件
add/set/remove,
hasNext/hasPrevious,
next/previous,
nextIndex,previousIndex,
可比較及比較器介面提供的方法:
java.lang.Comparable 可比較介面
compareTo
java.util.Comparator 比較器介面
compare/equals
method and call stack
程式呼叫方法(method)步驟:
1.進入方法前,先壓入返回位址(return address)到堆疊(call stack)
2.進入方法後,利用堆疊配置區域變數(local variable)空間
3.依序拷貝引數列(argument list)數值到參數列(parameter list)變數
4.開始執行方法內指令,直到遇到return指令或方法結束大括號}
5.離開方法前,釋放堆疊的區域變數空間
6.從堆疊彈出返回位址,從返回位址繼續執行指令
1.進入方法前,先壓入返回位址(return address)到堆疊(call stack)
2.進入方法後,利用堆疊配置區域變數(local variable)空間
3.依序拷貝引數列(argument list)數值到參數列(parameter list)變數
4.開始執行方法內指令,直到遇到return指令或方法結束大括號}
5.離開方法前,釋放堆疊的區域變數空間
6.從堆疊彈出返回位址,從返回位址繼續執行指令
methods for creating a self-contained .jar with data file
Given Main.java, data1.dat, data2.dat,
methods for creating a self-contained .jar with data file
0.use getResourceAsStream to get jar data at package root
InputStream is = Main.class.getResourceAsStream("/"+dataName);
1.add to all *.java source
package my.package;
2.compile with package start location
java -d . *.java
3.vi manifest.txt
#Class-Path: my.package
Main-Class: my.package.Main
4.produce jar file with data files
jar cvfm mypackage.jar manifest.txt my data1.dat data2.dat
5.run by
java -jar mypackage.jar
java -cp mypackage.jar my.package.Main
lowly and highly motivated
Keep lowly motivated to see how subtly things can go by themselves;
Keep highly motivated to see how far you can go by yourself.
常無欲以觀其妙;
常有欲以觀其徼.
參考:
1.http://www.shunto.org.hk/DaoDeiJinS/DaoDeiJin003.htm
「徼」有邊界、巡守之意:
Keep highly motivated to see how far you can go by yourself.
常無欲以觀其妙;
常有欲以觀其徼.
參考:
1.http://www.shunto.org.hk/DaoDeiJinS/DaoDeiJin003.htm
「徼」有邊界、巡守之意:
Evaluation Indices for Information Retrieval
資訊檢索評估指標計算法說明 2005/11/24-2008/12/8
==========================
假設某次查詢q1,資料集60篇中應有10篇相關文章,
但系統傳回15篇文章中,只有5篇屬相關文章,
現列出排名由高到低的15篇文章,如下,其中,+表相關,-表不相關,
d1,d2,d3,d4,d5,d6,d7,d8,d9,d10,d11,d12,d13,d14,d15
+, -, +, -, -, +, -, -, -, +, -, -, -, -, +
則就此次q1查詢,可計算如下各項評估指標,
機Y 機N .0 60 +1 59 -2 58 +3 57 -4 56 -5 55 +6 54 +15 45
------- ----- ---- ---- ---- ---- ---- ---- -----
人Y TP FN 0 10 1 9 1 9 2 8 2 8 2 8 3 7 5 5
人N FP TN 0 50 0 50 1 49 1 49 2 48 3 47 3 47 ... 10 40
a.一次查詢曲線結果
1.相關召回點準確率,P vs R, P@Rel
(註: P=TP/(TP+FP), R=TP/(TP+FN), FP=多進數, FN=少進數)
+ - + - - + - - - + - - - - +
R=0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.3 0.3 0.4 0.4 0.4 0.4 0.4 0.5
P=1/1 1/2 2/3 2/4 2/5 3/6 3/7 3/8 3/9 0.4 0.3 0.3 0.3 0.2 0.3
2.標準11召回點準確率,P vs R (0%,10%,20%,...,100%), P@11Rel
(註:當標準召回點無值或多值時,可用內插法,採計召回點以上最高準確率)
R=0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
P=1 1/1 2/3 3/6 0.4 0.3 0 0 0 0 0
3.標準3召回點準確率,P vs R (20%,50%,80%), P@3Rel
(註:當標準召回點無值或多值時,可用內插法,採計召回點以上最高準確率)
R=0.2 0.5 0.8
P=2/3 0.3 0
4.不相關出現點召回率(ROC接收器工作特性曲線),TP_rate vs FP_rate, ROC
(註:相關正判率 TP_rate=TP/(TP+FN),不相關誤判率 FP_rate=FP/(FP+TN))
+ - + - - + - - - + - - - - +
FP_rate=0/50 1/50 1/50 2/50 3/50 3/50 4/50 5/50 6/50 6/50 7/50 8/50 9/50 10/50 10/50
TP_rate=1/10 1/10 2/10 2/10 2/10 3/10 3/10 3/10 3/10 4/10 4/10 4/10 4/10 4/10 5/10
5.判別點準確率(lift chart提昇圖),TP vs subset_size, lift_chart
(註: subset_size = (TP+FP)/(TP+FP+TN+FN))
+ - + - - + - - - + - - - - +
subset_size=1/60 2/60 3/60 4/60 5/60 6/60 7/60 8/60 9/60 10/60 11/60 12/60 13/60 14/60 15/60
P=1 1 2 2 2 3 3 3 3 4 4 4 4 4 5
b.一次查詢單值結果
6.準確率(P@ALL), P=TP/(TP+NP),NP=多進數,不應進而進
P@ALL = 5 / (5 + 10)
7.召回率(R@ALL), R=TP/(TP+FP)=TP_rate,FP=少進數,應進而未進
R@ALL = 5 / (5 + 5)
8.相關出現點平均準確率, AveP@Rel
AveP@Rel = (1/1 + 2/3 + 3/6 + 4/10 + 5/15) / 5
9.總相關數召回點準確率, R-value, R-precision, P@R-value
P@R-value = 4/10
10.最大召回準確率之調和平均數(F measure),F=2/(1/R+1/P), maxF
+ - + - - + - - - + - - - - +
R=0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.3 0.3 0.4 0.4 0.4 0.4 0.4 0.5
P=1/1 1/2 2/3 2/4 2/5 3/6 3/7 3/8 3/9 0.4 0.3 0.3 0.3 0.2 0.3
F=0.18 0.16 0.3 0.28 0.26 0.37 0.35 0.33 0.31 0.4 0.38 0.36 0.34 0.33 0.4
maxF = 0.4
11.前10名準確率, P@10
P@10 = 4 / 10
12.前10名平均準確率, AveP@10,average precision)
AveP@10 = (1/1 + 1/2 + 2/3 + 2/4 + 2/5 + 3/6 + 3/7 + 3/8 + 3/9 + 4/10) / 10
13.成功率, (TP+TN) / (TP+FP+TN+FN), SUCCESS_RATE
SUCCESS_RATE = (5 + 40) / (5 + 10 + 40 + 5)
14.相關綜合判別率, TP_rate*(1-FP_rate) = TP*TN/(TP+FP)/(FP+TN)
Rel = 5 * 40 / (5+10) / (10 + 40)
假設有q1,q2,q3共計三次查詢,結果自資料集60篇文章中,
分別得到三次結果,如下,
應該傳回相關文章數(TP+FN),10,15,20,
實際傳回相關文章數(TP+FP),15,20,25,
其中,真正相關文章數(TP), 5, 2, 6,
詳情如下,
q1, +,-,+,-,-,+,-,-,-,+,-,-,-,-,+
q2, -,-,-,+,-,-,-,+,-,-,-,-,-,-,-,-,-,-,-,-
q3, -,+,-,-,+,-,+,-,-,-,+,-,-,-,+,-,-,-,-,-,+,-,-,-,-
機Y 機N q1 15 45 q2 20 40 q3 25 35
------- ----- ----- -----
人Y TP FN 5 5 2 13 6 14
人N FP TN 10 40 18 27 19 21
c.多次查詢曲線結果
15.召回點準確率平均圖,MP@Rel
仿照方法1,針對三次查詢,計算各召回點之準確率,再除以3,求平均
16.內插標準11召回點平均準確率, MP@11Rel
仿照方法2,針對三次查詢,計算11召回點之準確率,再除以3,求平均
17.無內插相關召回點平均準確率(依相關文章召回數及查詢數作平均)
q1_AveP@Rel = (1/1 + 2/3 + 3/6 + 4/10 + 5/15)/5
q2_AveP@Rel = (1/4 + 2/8)/2
q3_AveP@Rel = (1/2 + 2/5 + 3/7 + 4/11 + 5/10 + 6/16)/6
MAveP@Rel = (q1_AveP@Rel + q2_AveP@Rel + q3_AveP@Rel) / 3
18.特定相關數召回點平均準確率(前5,10,20,100相關文章召回點之查詢平均準確率)
q1_P@5R = 5/10
q2_P@5R = 無
q3_P@5R = 5/10
MP@5R = (q1_P@5R + q2_P@5R + q3_P@5R) / 3
19.總相關數召回點平均準確率(前5,10,20,100相關文章召回點之查詢平均準確率)
q1_P@R-value = 2/5
q2_P@R-value = 0/2
q3_P@R-value = 2/6
MP@5R = (q1_P@R-value + q2_P@R-value + q3_P@R-value) / 3
d.多次查詢單值結果
20.微觀平均準確率(micro average precision),依文章數作平均
map = (5 + 2 + 6) / (15 + 20 + 25)
21.巨觀平均準確率(macro average precision),依查詢數作平均
MAP = (5/15 + 2/20 + 6/25) / 3
22.前10名有正確之查詢比例 robust@10
q1前10名有正確文章=true
q1前10名有正確文章=true
q1前10名有正確文章=true
robust@10 = (1 + 1 + 1) / 3
23.總相關數召回點之查詢平均準確率, MP@R-value
q1總相關數召回點準確率=4/10
q2總相關數召回點準確率=2/10
q3總相關數召回點準確率=3/10
MP@R-value = (4/10 + 2/10 + 3/10) / 3
24.首相關排名之幾何平均數
q1首相關排名=1
q2首相關排名=4
q3首相關排名=2
幾何平均數=(1*4*2)^(1/3)
25.MAP(mean average precision),平均精確率平均
值介於0~1之間,越高越好,適用於二值相關度場合
MAP(Q) = 1/|Q| Sum_from_j=1_to_|Q| 1/m_j Sum_from_k=1_to_m_j Precision(R_jk)
Q:問題集, |Q|:問題數, m_j:問題j之應召回相關文件數,
R_jk: 問題j排名結果中,由前取到第k份相關文件出現為止的文件集
Precision(R_jk): 問題j第k份相關文件召回點之精確率
= 1/3 * [ 1/5 * (1/3 + 2/3 + 3/6 + 4/10 + 5/15)
+ 1/2 * (1/4 + 2/8)
+ 1/6 * (1/2 + 2/5 + 3/7 + 4/11 + 5/15 + 6/21)]
26.NDCG(normalized discounted cumulative gain),前k名累計打折正確率
值介於0~1之間,越高越好,適用於非二值相關度之機器學習場合
NDCG(Q,k)=1/|Q| Sum_from_j=1_to_|Q| Zk Sum_from_m=1_to_k [2^R(j,m) - 1]/log(1+m)
Q:問題集, |Q|:問題數,
Zk:正規化因子,讓NDCG在完美排名(前k份文件全相關)下,其值為1,
R(j,m): 問題j和文件m之相關度分數,介於0~1
若問題j回傳文件數k'小於指定的k,則NDCG(Q,k)就只累計到k'
例子: NDCG(Q,k=2)
= 1/3 * { Z2 * [(2^1-1)/log(1+1) + (2^0-1)/log(1+2)]
+ Z2 * [(2^0-1)/log(1+1) + (2^0-1)/log(1+2)]
+ Z2 * [(2^0-1)/log(1+1) + (2^1-1)/log(1+2)]}
with Z2 = (2^1-1)/log(1+1) + (2^1-1)/log(1+2)
e.多次查詢表格結果
25.查詢問題數,所有查詢傳回文章總數
查詢問題數=3
所有查詢傳回文章總數=60
26.所有查詢傳回相關文章總數,所有查詢實際相關文章總數
所有查詢傳回相關文章總數=15+20+25
所有查詢實際相關文章總數=5+2+6
參考文獻
1.baezayates-aw-99-modern information retrieval
2.witten-mkp-99-managing gigabytes
3.witten-mkp-00-data mining- practical machine learning tools and techniques
4.manning-cup-08-introduction to information retrieval
how to read big5 files in Java
/*
ReadBig5File.java
read a file in big5 code
> javac ReadBig5File.java
> java ReadBig5File big5.txt
Big5編碼文字檔
*/
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.io.BufferedReader;
public class ReadBig5File
{
public static void main(String args[]) throws
java.io.FileNotFoundException,
java.io.UnsupportedEncodingException,
java.io.IOException
{
String file="big5.txt";
FileInputStream fis = new FileInputStream(new File(file));
// java.io.FileNotFoundException
BufferedReader br =new BufferedReader(new InputStreamReader(fis,"BIG5"));
// java.io.UnsupportedEncodingException
// java.io.IOException
while(br.ready())
{
String line=br.readLine();
System.out.println(line);
System.out.flush();
}
br.close();
}
}
how to get unicode/big5 code in java String
/*
ConvertBig5Unicode.java
print string in big5 and unicode
>javac ConvertBig5Unicode.java
>java ConvertBig5Unicode
查詢字串: 今天天氣很好
查詢字串的統一碼 : 4eca 5929 5929 6c23 5f88 597d
查詢字串的大五碼 : a4b5 a4d1 a4d1 aef0 abdc a66e
*/
import java.io.File;
// 如何取得java字串之unicode及big5碼
public class ConvertBig5Unicode
{
public static void main(String args[]) throws java.io.UnsupportedEncodingException
{
String query="今天天氣很好";
String unicode_query="";
for(int i=0; i<=query.length()-1;i++)
{
if(query.charAt(i)>=0x100)
unicode_query += Integer.toHexString(query.charAt(i)) + " ";
else
unicode_query += query.charAt(i);
}
byte [] big5_stream=query.getBytes("big5");
// java.io.UnsupportedEncodingException
String big5_query="";
for(int i=0; i<=big5_stream.length-1;i++)
{
int b1,b2;
b1 = big5_stream[i];
b2 = (i+1<=big5_stream.length-1)? big5_stream[i+1] : 0;
b1 = (b1>=0) ? b1 : 0x100+b1;
b2 = (b2>=0) ? b2 : 0x100+b2;
//out.println("b1="+b1+",b2="+b2);
if((b1 >= 0xa1 && b1 <= 0xf9) && ((b2 >= 0x40 && b2 <= 0x7e) || (b2 >= 0xa1 && b2 <= 0xfe)))
{
big5_query += Integer.toHexString(b1) + Integer.toHexString(b2) + " ";
i++;
}
else
big5_query += (char) big5_stream[i];
}
System.out.println("查詢字串: " + query);
System.out.println("查詢字串的統一碼 : "+unicode_query);
System.out.println("查詢字串的大五碼 : "+big5_query);
}
}
dataparksearch檢索系統安裝摘要
dataparksearch+mysql安裝摘要 2008.11.20.
----------------------------
http://www.dataparksearch.org/#download
http://www.dataparksearch.org/dpsearch-4.50.tar.bz2
http://www.dataparksearch.org/add-on/TraditionalChinese.freq.gz
http://www.appservnetwork.com/
http://prdownloads.sourceforge.net/appserv/appserv-win32-2.6.0.exe?download
1.解壓編譯佈署
# 解開壓縮及包裝
bunzip2 -c dpsearch-4.50.tar.bz2 | tar tvf -
#產生編譯指令檔Makefile
cd dpsearch-4.50/
perl install.pl
...
Enable support for extra charsets? yes
...
或
./configure
--prefix=/home/me/dataparksearch
--bindir=/usr/local/dpsearch/bin
--sbindir=/usr/local/dpsearch/sbin
--sysconfdir=/usr/local/dpsearch/etc
--localstatedir=/usr/local/dpsearch/var
--libdir=/usr/local/dpsearch/lib
--includedir=/usr/local/dpsearch/include
--mandir=/usr/local/dpsearch/man
--enable-shared
--disable-syslog
--enable-pthreads
--enable-parser
--enable-mp3
--without-aspell
--with-extra-charsets=all
--enable-file
--enable-http
--enable-ftp
--enable-htdb
--enable-news
--with-mysql=/usr/local/mysql/include
#產生執行檔indexer,search.cgi等
dpsearch-4.50/make
#佈署組態檔.conf及執行檔indexer,search.cgi等
dpsearch-4.50/make install
#解壓佈署字典檔
gunzip TraditionalChinese.freq.gz #解開壓縮
cp TraditionalChinese.freq /usr/local/dpsearch/etc/ #安裝中文字典
2.設定索引組態檔
cd /usr/loca/dpsearch/etc/
cp indexer.conf-dist indexer.conf #indexer使用
vi indexer.conf
DBAddr mysql://username:password@localhost/dpsearch/?dbmode=single
LocalCharset BIG5 #資料庫字集
RemoteCharset BIG5 #索引來源網站字集
LoadChineseList BIG5 TraditionalChinese.freq #中文字典位置
#Include langmap.conf #載入太多字集,可能識別錯誤,宜用下行,只載入單一字集
LangMapFile langmap/zh.big5.lm
Server site http://your.server.name/path 索引網站下所有網頁
Server path http://your.server.name/path 索引路徑下所有網頁
Server page http://your.server.name/path 索引單一網頁
#cp langmap.conf-dist langmap.conf
#vi langmap.conf
# LangMapFile langmap/zh.big5.lm
cp search.htm-dist search.htm #search.cgi使用
vi search.htm
DBAddr mysql://user:password@localhost/dpsearch/?dbmode=single
LocalCharset big5 #資料庫字集
BrowserCharset big5 #顯示用字集
cp stopwords.conf-dist stopwords.conf
cp sections.conf-dist sections.conf
3.建立空白索引資料庫
#啟動mysql資料庫服務
service mysql start
#建立dpsearch資料庫
/usr/bin/mysqladmin -u root -p create dpsearch
Enter password:
#設定dpsearch資料庫供用戶user及密碼password使用
/usr/bin/mysql -u root -p dpsearch
Enter password:
mysql> grant all privileges on dpsearch.* to user@localhost identified by 'password';
mysql> flush privileges;
mysql> quit
4.建立索引資料表大綱及內容
/usr/local/dpsearch/sbin/indexer -Edrop #清除舊大綱
/usr/local/dpsearch/sbin/indexer -Ecreate #建立新大綱
/usr/local/dpsearch/sbin/indexer -a #建立索引
/usr/local/dpsearch/sbin/indexer -S #索引狀態
5.耙取網頁建立索引資料庫
/usr/local/dpsearch/sbin/indexer -Edrop #清除舊大綱
/usr/local/dpsearch/sbin/indexer -Ecreate #建立新大綱
#依據index.conf的Server指示起始位置,耗取網頁,建立索引
/usr/local/dpsearch/sbin/indexer -a -v 5 #詳細訊息顯示索引過程
6.利用索引資料庫檢索網頁
#啟動apache網頁伺服器服務
service http start
#建立檢索首頁,假設網頁伺服器首頁cgi路徑為/var/www/cgi-bin/
cp /usr/local/dpsearch/bin/search.cgi /var/www/cgi-bin/
#利用瀏覽器開啟檢索首頁
firefox http://localhost/cgi-bin/search.cgi
7.中文索引問題
a.網頁字集判別錯誤,可依下法知道是否判別錯誤,
/usr/local/dpsearch/sbin/dpguesser [-n maxhits] < web.htm
444h 214m zh Big5
58h 235m zh GB2312
118h 235m es ISO-8859-1
136h 235m ja UTF-8
或
/usr/local/dpsearch/sbin/indexer -v 5
indexer[16184]: {01} Guesser: Lang: zh, Charset: Big5
解決法為減少自動識別的字集,最好indexer.conf只用一種字集即可,如下,
#Include langmap.conf
LangMapFile langmap/zh.big5.lm
b.參考http://www.dataparksearch.org/dpsearch-international.en.html
註:
1.以上資料部份參考mikanagi-tku-06-search_engines gais and dataparksearch.ppt
2.dataparksearch計畫來自mnogosearch(www.mnogosearch.org)計畫,兩者差異如下,
dataparksearch: gpl, sql/cache, linux
mnogosearch: gpl, sql, windows, shareware
訂閱：
文章 (Atom)
how to connect codex app to remote ollama backend through caddy relay
如何讓 Codex App 連接公司架設的 LLM 從 Caddy 閘道安裝、遠端 Ollama API 金鑰，到 Codex 的 config.toml 組態設定，一次整理成可直接照做的流程。 前言 OpenAI 提供免費下載...